User Tools

Site Tools


extensions:ldap-data-collector

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
extensions:ldap-data-collector [2019/11/18 16:32]
vdumas
extensions:ldap-data-collector [2020/07/07 18:08] (current)
Line 4: Line 4:
 description_wiki : Inventory Data Collector for LDAP description_wiki : Inventory Data Collector for LDAP
 index_hidden ​    : yes index_hidden ​    : yes
-version ​         : 1.2.0 +version ​         : 1.2.2 
-release_dt ​      : ​2018-07-24+release_dt ​      : ​2020-07-07
 itop-version-min :  itop-version-min : 
 dependencies_s ​  : ​ dependencies_s ​  : ​
Line 11: Line 11:
 download_hidden ​ :  download_hidden ​ : 
 code             : ldap-data-collector code             : ldap-data-collector
-module-lists_hidden : itop-data-collector-base/​1.0.12,​ itop-data-collector-vsphere/1.0.1+module-lists_hidden : itop-data-collector-base/​1.0.12,​ itop-data-collector-ldap/1.0.2
 standalone ​      : yes standalone ​      : yes
 ---- ----
Line 37: Line 37:
 ===== Revision History ===== ===== Revision History =====
 ^  Version ​ ^  Release Date  ^  Comments ​ ^ ^  Version ​ ^  Release Date  ^  Comments ​ ^
 +|  2020-07-07 ​ |  1.2.2  | Support of LDAP URI scheme for the connection,​\\ Better debug information via ldap-test.php,​\\ Configurable target class to create either users of type UserLDAP or UserExternal for example.\\ Request only the needed attributes (and explicitely request memberof)\\ Additional command line parameters for ''​ldap_test.php''​\\ Multi configuration file\\ New CSV collector\\ Configurable timestamp added in the logs\\ New option for usage: –help|
 +|  2020-02-17 ​ |  1.2.1  | Never publicly released, only updates to data collector base.\\ Fix "​undefined constant TABLENAME_PATTERN"​\\ Reject invalid characters for database_table_name\\ Performance enhancement:​ retrieve only the needed fields when performing a lookup\\ Added the specific class MySQLCollector which forces the DB connection to use UTF-8 characters |
 |  2018-08-28 ​ |  1.2.0  | First public release on iTopHub, refactoring of the code and configuration parameters. | |  2018-08-28 ​ |  1.2.0  | First public release on iTopHub, refactoring of the code and configuration parameters. |
 |  2017-06-22 ​ |  1.1.1  | Version to use latest version of collector-base | |  2017-06-22 ​ |  1.1.1  | Version to use latest version of collector-base |
Line 42: Line 44:
 |  2015-05-07 ​ |  1.0.0  | Initial version | |  2015-05-07 ​ |  1.0.0  | Initial version |
  
 +    ​
  
 ===== Limitations ===== ===== Limitations =====
Line 54: Line 57:
   * An LDAP access to the Enterprise directory and a read user to access the data.   * An LDAP access to the Enterprise directory and a read user to access the data.
   * An HTTP/HTTPS access to the iTop web services (REST + synchro_import.php and synchro_exec.php)   * An HTTP/HTTPS access to the iTop web services (REST + synchro_import.php and synchro_exec.php)
 +  * + [[extensions:​itop-data-collector-base#​requirements|Data collector Base]] requirements.
  
 ===== Installation ===== ===== Installation =====
Line 62: Line 66:
 By default this file should contains the values used to connect to the LDAP server and to the iTop server: By default this file should contains the values used to connect to the LDAP server and to the iTop server:
  
-<code xml>+<file xml params.local.xml>​ 
 +<?xml version="​1.0"​ encoding="​UTF-8"?>​ 
 +<!-- conf/​params.local.xml - your specific configuration parameters -->
 <​parameters>​ <​parameters>​
   <​itop_url>​http://​localhost/</​itop_url>​   <​itop_url>​http://​localhost/</​itop_url>​
Line 69: Line 75:
   <​contact_to_notify>​john.doe@demo.com</​contact_to_notify>​   <​contact_to_notify>​john.doe@demo.com</​contact_to_notify>​
   <​synchro_user>​admin</​synchro_user>​   <​synchro_user>​admin</​synchro_user>​
-  <ldaphost>​localhost</​ldaphost+  <ldapuri>ldap://localhost:389</​ldaphost>​
-  <​ldapport>​389</​ldapport>+
   <​ldapdn>​DC=company,​DC=com</​ldapdn>​   <​ldapdn>​DC=company,​DC=com</​ldapdn>​
   <​ldaplogin>​CN=ITOP-LDAP,​DC=company,​DC=com</​ldaplogin>​   <​ldaplogin>​CN=ITOP-LDAP,​DC=company,​DC=com</​ldaplogin>​
Line 82: Line 87:
   <​json_placeholders>​   <​json_placeholders>​
     <​full_load_interval>​604800</​full_load_interval><​!-- 7 days (in seconds): 7*24*60*60 -->     <​full_load_interval>​604800</​full_load_interval><​!-- 7 days (in seconds): 7*24*60*60 -->
 +    <​users_target_class>​UserLDAP</​users_target_class>​
     <​synchro_status>​production</​synchro_status>​     <​synchro_status>​production</​synchro_status>​
   </​json_placeholders>​   </​json_placeholders>​
 </​parameters>​ </​parameters>​
-</code>+</file>
  
 ^ Parameter ^ Meaning ^ Sample value ^ ^ Parameter ^ Meaning ^ Sample value ^
Line 93: Line 99:
 | contact_to_notify | The email address of an existing contact in iTop, to be notified in case of error during the synchronization |john.doe@demo.com | | contact_to_notify | The email address of an existing contact in iTop, to be notified in case of error during the synchronization |john.doe@demo.com |
 | synchro_user | iTop user set as allowed to run synchronization. It is highly recommended to use the same as itop_login | admin | | synchro_user | iTop user set as allowed to run synchronization. It is highly recommended to use the same as itop_login | admin |
-| ldaphost| ​The address to connect to LDAP server ​| localhost | +| ldaphost| ​**obsolete**,​ Use ''​ldapuri''​ instead. ​| localhost | 
-| ldapport| ​TCP port | 389|+| ldapport| ​**obsoelete**,​ use ''​ldapuri''​ instead. ​| 389|
 | ldapdn| Company DN for LDAP|DC=company,​DC=com | | ldapdn| Company DN for LDAP|DC=company,​DC=com |
 | ldaplogin| Login to connect to LDAP server|CN=ITOP-LDAP,​DC=company,​DC=com| | ldaplogin| Login to connect to LDAP server|CN=ITOP-LDAP,​DC=company,​DC=com|
 | ldappassword| Password to connect to LDAP server | | | ldappassword| Password to connect to LDAP server | |
 +| ldapuri| The URI to connect to the LDAP server, either <​nowiki>​ldap://<​host>:<​port>​ or ldaps://<​host>:<​port></​nowiki>​ | |
 | prefix| A unique string for each LDAP server. MUST be non-empty if you run several instances of the collector against the same iTop instance. Can contain only [a-z0-9_] characters. | | | prefix| A unique string for each LDAP server. MUST be non-empty if you run several instances of the collector against the same iTop instance. Can contain only [a-z0-9_] characters. | |
 | full_load_interval | Duration (in seconds) for which to retain records not found in LDAP. | | | full_load_interval | Duration (in seconds) for which to retain records not found in LDAP. | |
 | synchro_status| For information:​ the status of the Synchronization Data Sources (production,​ implementation or obsolete) | production | | synchro_status| For information:​ the status of the Synchronization Data Sources (production,​ implementation or obsolete) | production |
 +| users_target_class | The class of User objects to create in iTop when synchronizing users. Either UserLDAP or UserExternal | UserLDAP |
  
 <note warning>​Starting with iTop version 2.5.0, the account used to connect to iTop **must have** the profile ''​REST Services user''​ in order to be allowed to use the web services.</​note>​ <note warning>​Starting with iTop version 2.5.0, the account used to connect to iTop **must have** the profile ''​REST Services user''​ in order to be allowed to use the web services.</​note>​
Line 212: Line 220:
 ===== Troubleshooting ===== ===== Troubleshooting =====
  
-You can test your configuration **without importing any data in iTop** by running the following command from the command line:+==== Connection problems ==== 
 + 
 +To test and troubleshoot connection problems, use the script ''​ldap-test.php''​ located in the ''​collector/​bin''​ folder. 
 +The script uses the same parameters as the normal collector, but produces more debug output. 
 + 
 +So edit the configuration in the file ''​conf/​params.local.xml''​ then launch the test script by typing the following command from the command prompt. 
 + 
 +<​code>​ 
 +php collectors/​bin/​ldap-test.php 
 +</​code>​ 
 + 
 +If you see a message like: 
 +<​code>​ 
 +Error - ldap_bind('​cn=admin,​dc=combodo,​dc=com',​ '​*******'​) FAILED (Can't contact LDAP server). 
 +</​code>​ 
 + 
 +then something is wrong with the connection to the LDAP server. 
 + 
 +  - Check that parameter ''<​ldapuri>''​ is correct. (protocol, host and port) 
 +  - Check that the connection to the server is not blocked by a firewall (You can use the command ''​telnet <​host>​ <​port>''​ and see if the connection is established). 
 +  - Check for TSL/SSL problems. If you see the following text in the output of the ''​ldap-test.php''​ script, then the problem is likely related to a TLS certificate:​ 
 +<​code>​ 
 +attempting to connect:  
 +connect success 
 +TLS: peer cert untrusted or revoked (0x402) 
 +TLS: can't connect: (unknown error code). 
 +</​code>​ 
 + 
 +The solution is to instruct LDAP to ignore this faulty certificate,​ by adding the following lines to the //LDAP configuration file// (see the note below). 
 +<​code>​ 
 +# Ignore the server'​s certificate 
 +TLS_REQCERT never 
 +</​code>​ 
 + 
 +<note tip>On Linux systems; the OpenLDAP library used by PHP tries to load successively the following configuration files: 
 +  - /​etc/​ldap/​ldap.conf 
 +  - /​home/<​current_user>/​ldaprc 
 +  - /​home/<​current_user>/​.ldaprc 
 +  - <​current_folder>/​ldaprc 
 + 
 +You can put the above mentioned parameter in any of the files, but be aware that the first file (/​etc/​ldap/​ldap.conf) affects the whole system, whereas the other configuration files affect scripts running under the current user, or only scripts ran from the current directory. 
 + 
 +The syntax for all thoses files is the same. For more information,​ refer to: [[https://​www.openldap.org/​software/​man.cgi?​query=ldap.conf|ldap.conf man page]] 
 +</​note>​ 
 + 
 +==== Data collection problems ==== 
 + 
 +If the output of the ''​ldap-test.php''​ script contains: 
 +<​code>​ 
 +Error - ldap_search('​dc=combodo,​dc=net',​ '​(objectClass=inetOrgPerson)'​) FAILED (No such object). 
 +</​code>​ 
 + 
 +Then check the LDAP query used for retrieving the "​contacts"​. This query is defined by the two parameters:​ 
 +<code xml> 
 +    <​ldapdn>​DC=company,​DC=com</​ldapdn>​ 
 +     
 +    <!-- Parameters for Person synchronization --> 
 +    <​ldappersonfilter>​(objectClass=person)</​ldappersonfilter>​ 
 +</​code>​ 
 + 
 + 
 +If the LDAP query is correct, you should see an output similar to: 
 + 
 +<​code>​ 
 +List of the attributes to retrieve (taken from the mapping): 
 +uid,​sn,​givenname,​mail,​telephonenumber,​mobile,​title,​employeenumber,​memberof 
 +Use --attributes=x,​y,​z to retrieve x, y and z instead. Use --attributes=* to retrieve all fields. 
 +Debug - ldap_connect('​ldaps://​customers.combodo.com'​)... 
 +Debug - ldap_bind('​cn=admin,​dc=combodo,​dc=com',​ '​c8mb0do'​)... 
 +Debug - ldap_bind() Ok. 
 +Debug - ldap_search('​dc=combodo,​dc=com',​ '​(objectClass=inetOrgPerson)',​ ['​uid',​ '​sn',​ '​givenname',​ '​mail',​ '​telephonenumber',​ '​mobile',​ '​title',​ '​employeenumber',​ '​memberof'​])... 
 +Debug - ldap_search() Ok. 
 +The LDAP query '​(objectClass=inetOrgPerson)'​ returned 13 elements. 
 +Displaying only 10 elements (use --max-records=xx to change this limit). 
 +------------------------------------------------ 
 +LDAP Structure:​ 
 +Info: when a field is empty on a given record, it is not returned by LDAP. 
 +------------------------------------------------ 
 +givenname : bruce 
 +sn        : Lee 
 +uid       : blee 
 +mail      : bruce.lee2@combodo.com 
 +mobile ​   : 0608080808 
 +------------------------------------------------ 
 +givenname : chuck 
 +mail      : chuck.norris@combodo.com 
 +sn        : Norris 
 +uid       : cnorris 
 +------------------------------------------------ 
 +</​code>​ 
 + 
 +The first column of the output is the name of the field in LDAP (all fields returned by the LDAP query are listed) and the second column shows the values of the first record found in LDAP. 
 +Based on the values displayed you can complete the configuration of the mapping in the configuration file ''​conf/​params.local.xml''​. 
 + 
 +<note tip>By default ''​ldap_test.php''​ only requests the attributes used in the Person'​s mapping. To request all the available LDAP attributes, add the parameter ''<​nowiki>​--attributes=*</​nowiki>''​ to the ''​ldap_test.php''​ command line</​note>​ 
 + 
 +<note tip>By default ''​ldap_test.php''​ dumps only the first 10 records of the results. You can adjust this number to //xx// records by specifying the parameter ''<​nowiki>​--max-records=xx</​nowiki>''​ on the command line.</​note>​ 
 + 
 + 
 +Finally you can test your configuration **without importing any data in iTop** by running the following command from the command line:
 <​code>​php exec.php --console_log_level=9 --collect_only</​code>​ <​code>​php exec.php --console_log_level=9 --collect_only</​code>​
  
Line 248: Line 355:
   <​ldapdn>​OU=FGA,​DC=combodo,​DC=net</​ldapdn>​   <​ldapdn>​OU=FGA,​DC=combodo,​DC=net</​ldapdn>​
   <​ldaplogin>​COMBODO\administrateur</​ldaplogin>​   <​ldaplogin>​COMBODO\administrateur</​ldaplogin>​
-  <​ldappassword>​c8mb0doSARL</​ldappassword>​+  <​ldappassword>​xxxxxx</​ldappassword>​
   <​ldappersonfilter>​(objectClass=person)</​ldappersonfilter>​   <​ldappersonfilter>​(objectClass=person)</​ldappersonfilter>​
   <​itop_group_pattern>/​^CN=itop-(.*),​OU=.*/</​itop_group_pattern>​   <​itop_group_pattern>/​^CN=itop-(.*),​OU=.*/</​itop_group_pattern>​
Line 312: Line 419:
  
  
-If you are unsure of the mapping between the fields in your LDAP/AD server and iTop you can use the script ''​ldap-test.php''​ provided with the connector to dump the result of the LDAP query. 
  
-Once the connection to the LDAP server is properly configured on your ''​conf/​params.local.xml''​ file, run the follwing script for the command line: 
- 
-<​code>​php collectors/​bin/​ldap-test.php</​code>​ 
- 
-This produces the following output: 
-<​code>​ 
-The LDAP query '​(objectClass=person)'​ returned 345 elements. 
------------------------------------------------- 
-LDAP Structure: 
------------------------------------------------- 
-objectclass ​          : top 
-                        person 
-                        organizationalPerson 
-                        user 
-cn                    : John Doe 
-sn                    : Doe 
-givenname ​            : John 
-distinguishedname ​    : CN=John Doe,​OU=US,​OU=Users,​DC=demo,​DC=com 
-instancetype ​         : 4 
-whencreated ​          : 20120105094632.0Z 
-whenchanged ​          : 20120124113752.0Z 
-displayname ​          : John Doe 
-usncreated ​           : 12749 
-memberof ​             : CN=itop-Portal user,​DC=demo,​DC=com 
-                        CN=Employees,​CN=Users,​DC=demo,​DC=com 
-usnchanged ​           : 16519 
-name                  : John Doe 
-objectguid ​           : w~��~�@�˟ϻ�[k 
-useraccountcontrol ​   : 66048 
-badpwdcount ​          : 0 
-codepage ​             : 0 
-countrycode ​          : 0 
-badpasswordtime ​      : 129706868110312500 
-lastlogoff ​           : 0 
-lastlogon ​            : 129706877712031250 
-pwdlastset ​           : 129702303923125000 
-primarygroupid ​       : 513 
-objectsid ​            : oC 
-,​00L��H}�R 
-accountexpires ​       : 9223372036854775807 
-logoncount ​           : 0 
-samaccountname ​       : johndoe 
-samaccounttype ​       : 805306368 
-userprincipalname ​    : john.doe@demo.com 
-objectcategory ​       : CN=Person,​CN=Schema,​CN=Configuration,​DC=demo,​DC=com 
-dscorepropagationdata : 20120124113752.0Z 
-                        20120123183028.0Z 
-                        20120123161623.0Z 
-                        16010101181216.0Z 
-lastlogontimestamp ​   : 129718175750781250 
-mail                  : john.doe@demo.com 
-</​code>​ 
- 
-The first column of the output is the name of the field in LDAP (all fields returned by the LDAP query are listed) and the second column shows the values of the first record found in LDAP. 
-Based on the values displayed you can complete the configuration of the mapping in the configuration file ''​conf/​params.local.xml''​. 
  
 ===== Usage ===== ===== Usage =====
  
-To launch the data collection and synchronization with iTop, run the following command (from the root directory where the data collector application is installed):​ +{{section>extensions:itop-data-collector-base#​usage&​nouser&​nodate&​nofooter&​noindent&​noheader}}
- +
-<code> +
-php exec.php +
-</​code>​ +
- +
-The following (optional) command line options are available: +
- +
-^ Option ^ Meaning ^ default value ^ +
-| <​nowiki>​--console_log_level=<​level></​nowiki>​ | Level of output to the console. From -1 (none) to 9 (debug). | 6 (info) | +
-| <​nowiki>​--collect_only</​nowiki>​ | Run only the data collection, but do not synchronize the data with iTop | false | +
-| <​nowiki>​--synchro_only</​nowiki>​ | Synchronizes the data previously collected (stored in the ''​data''​ directory) with iTop. Do not run the collection. | false | +
-| <​nowiki>​--configure_only</​nowiki>​ | Check (and update if necessary) the synchronization data sources in iTop and exit. Do NOT run the collection or the synchronization | +
-| <​nowiki>​--max_chunk_size=<​size></​nowiki>​ | Maximum number of items to process in one pass, for preserving the memory of the system. If there are more items to process, the application will iterate. | 1000 |+
  
 The execution of the command line will: The execution of the command line will:
extensions/ldap-data-collector.1574091161.txt.gz · Last modified: 2019/11/18 16:32 (external edit)

";