User Tools

Site Tools


extensions:itop-data-collector-base

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
extensions:itop-data-collector-base [2019/02/05 18:14]
vdumas [Data Collector Base]
extensions:itop-data-collector-base [2019/11/12 10:27]
vdumas [Data Collector Base]
Line 1: Line 1:
-====== Data Collector Base ======+======= Data Collector Base ======
 ---- dataentry summary ---- ---- dataentry summary ----
 name             : Data Collector Base name             : Data Collector Base
 description_wiki : Inventory Data Collector toolkit for creating your own data collectors for iTop description_wiki : Inventory Data Collector toolkit for creating your own data collectors for iTop
 index_hidden ​    : yes index_hidden ​    : yes
-version ​         : 1.0.11 +version ​         : 1.1.2 
-release_dt ​      : ​2018-06-26+release_dt ​      : ​2019-11-07
 itop-version-min :  itop-version-min : 
-dependencies_s ​  :  +download_url ​    : http://​www.combodo.com/​itop-extensions/​itop-data-collector-base-1.1.2-496.zip
-diffusion ​       : iTop Hub, Combodo site +
-download_url ​    : http://​www.combodo.com/​itop-extensions/​itop-data-collector-base-1.0.11-346.zip +
-module-lists_hidden : itop-data-collector-base/​1.0.11+
 ---- ----
  
Line 37: Line 34:
 ===== Revision History ===== ===== Revision History =====
 ^  Release Date  ^  Version ​ ^  Comments ​ ^ ^  Release Date  ^  Version ​ ^  Comments ​ ^
 +|  2019-11-07 ​ |  1.1.2  | Fix "​undefined constant TABLENAME_PATTERN"​ |
 +|  2019-10-28 ​ |  1.1.1  | Contains upgrades from both 1.0.13 and 1.1.0 \\ * Reject invalid characters for database_table_name |
 +|  2019-10-28 ​ |  1.1.0  | Based on 1.0.9 \\ * Added the specific class MySQLCollector which forces the DB connection to use UTF-8 characters |
 +|  2019-10-28 ​ |  1.0.13 ​ | * LookupTables can now be non case sensitive (since MySQL is not) \\         * Prevent a warning in SQLCollector for each "​ignored"​ attribute \\         * Improved support of iTop 2.4+ (obsolescence flag) |
 +|  2019-10-28 ​ |  1.0.12 ​ | * removed a warning in PHP 7.2 |
 |  2018-06-26 ​ |  1.0.11 ​ | Added a debug trace (visible if console_log_level=9) to show which mapping regular expression is applied (when one is applied).\\ Bug fix: properly handle utf-8 characters in the mapping table'​s regular expressions (/u modifier)\\ Make the cUrl/SSL options configurable to suit all possible combinations and security considerations. | |  2018-06-26 ​ |  1.0.11 ​ | Added a debug trace (visible if console_log_level=9) to show which mapping regular expression is applied (when one is applied).\\ Bug fix: properly handle utf-8 characters in the mapping table'​s regular expressions (/u modifier)\\ Make the cUrl/SSL options configurable to suit all possible combinations and security considerations. |
 |  2015-06-30 ​ |  1.0.10 ​ | New class of collector: ''​MySQLCollector''​ which forces the retrieved data to be encoded in UTF-8.| |  2015-06-30 ​ |  1.0.10 ​ | New class of collector: ''​MySQLCollector''​ which forces the retrieved data to be encoded in UTF-8.|
Line 68: Line 70:
 ===== Configuration ===== ===== Configuration =====
  
-At minimum, edit the file ''​params.local.xml'' ​and adjust ​the following ​values:+''​params.local.xml'' ​is the only file to edit to configure a collector. \\ 
 + 
 +At minimum ​the following ​parameters must be set in this file:
  
 <code xml> <code xml>
Line 91: Line 95:
 | stop_on_synchro_error | Whether or not to stop when an error occurs during a synchronization (''​yes''​ or ''​no''​). | no | | stop_on_synchro_error | Whether or not to stop when an error occurs during a synchronization (''​yes''​ or ''​no''​). | no |
 | console_log_level | Level of ouput to the console. From -1 (none) to 9 (debug). | 6 (info) | | console_log_level | Level of ouput to the console. From -1 (none) to 9 (debug). | 6 (info) |
 +| curl_options | When using cUrl to connect to the iTop Webservices the cUrl options can be specified in this section. The syntax is <​CURLOPT_NAME_OF_THE_OPTION1>​VALUE_1</​CURLOPT_NAME_OF_THE_OPTION1>​ where VALUE_x are either:\\ The numeric value of the option,\\ or the string representation of the corresponding PHP "​define"​ (case sensitive).\\ It is possible to define several php_curl options like in the example below| ​ |
  
 +<note tip>
 +<​code>​
 +<​curl_options>​
 +    <​CURLOPT_SSLVERSION>​CURL_SSLVERSION_SSLv3</​CURLOPT_SSLVERSION>​
 +    <​CURLOPT_SSL_VERIFYHOST>​0</​CURLOPT_SSL_VERIFYHOST>​
 +    <​CURLOPT_SSL_VERIFYPEER>​1</​CURLOPT_SSL_VERIFYPEER>​
 +  </​curl_options>​
 +</​code>​
 +</​note>​
  
 <note tip>The file ''​params.distrib.xml''​ contains the default values for the parameters. Both files (''​params.distrib.xml''​ and ''​params.local.xml''​) use exactly the same format. But ''​params.distrib.xml''​ is considered as the reference and should remain unmodified. Should you need to change the value of a parameter, copy and modify its definition in ''​params.local.xml''​. The values in ''​params.local.xml''​ have precedence over the ones in ''​params.distrib.xml''</​note>​ <note tip>The file ''​params.distrib.xml''​ contains the default values for the parameters. Both files (''​params.distrib.xml''​ and ''​params.local.xml''​) use exactly the same format. But ''​params.distrib.xml''​ is considered as the reference and should remain unmodified. Should you need to change the value of a parameter, copy and modify its definition in ''​params.local.xml''​. The values in ''​params.local.xml''​ have precedence over the ones in ''​params.distrib.xml''</​note>​
Line 127: Line 141:
  
 <code php> <code php>
-Orchestrator::​AddRequirement('​5.4.0'​);​ //​This ​code requires at least PHP 5.4 +Orchestrator::​AddRequirement('​5.4.0'​);​ //This requires at least PHP 5.4 
-Orchestrator::​AddRequirement('​1.2.0',​ '​ldap'​);​ //​This ​code requires the ldap extension, with at least version 1.2.0+Orchestrator::​AddRequirement('​1.2.0',​ '​ldap'​);​ //This requires ​at least the ldap extension version 1.2.0
 </​code>​ </​code>​
 ==== Creating the JSON definition file ==== ==== Creating the JSON definition file ====
 The simpler way to create the JSON file for a Synchro Data Source, is to export the definition of an existing data source. The simpler way to create the JSON file for a Synchro Data Source, is to export the definition of an existing data source.
   * Create the synchronisation data source in iTop, adjust its parameters (attributes,​ etc.) to suit your needs   * Create the synchronisation data source in iTop, adjust its parameters (attributes,​ etc.) to suit your needs
-  * Use the command line tool ''​dump_task.php''​ (available in the ''​toolkit''​ folder to produce the JSON file:+  * Use the command line tool ''​dump_tasks.php''​ (available in the ''​toolkit''​ folder to produce the JSON file:
  
-<​code>​php toolkit/dump_task.php --task_name="​name of the task to export"​ > collectors/​myCollector.json</​code>​+<​code>​php toolkit/dump_tasks.php --task_name="​name of the task to export"​ > collectors/​myCollector.json</​code>​
  
 Inside your data source definition you can use special placeholders to make the data source configurable by the user of the application,​ or to adjust its behavior via some special settings: Inside your data source definition you can use special placeholders to make the data source configurable by the user of the application,​ or to adjust its behavior via some special settings:
Line 152: Line 166:
 <code xml params.local.xml>​ <code xml params.local.xml>​
 <?xml version="​1.0"​ encoding="​UTF-8"?>​ <?xml version="​1.0"​ encoding="​UTF-8"?>​
-<​!-- ​ Local values for parameters. The values defined in this file have precedence over the ones defined in params.distrib.xml -->+<​!-- ​ Local values for parameters. ​--> 
 +<​!--  ​The values defined in this file have precedence over the ones defined in params.distrib.xml -->
 <​parameters>​ <​parameters>​
   <​itop_url>​https://​localhost/​trunk</​itop_url>​   <​itop_url>​https://​localhost/​trunk</​itop_url>​
Line 514: Line 529:
 </​note>​ </​note>​
  
 +<note warning>
 +If you modify the list of fields of a class already collected, you **must** update your JSON definition file, to specify what to do with the new fields. By default they are added to existing Data Synchro with ''​no update''​ and ''​no lock''​.
 +</​note>​
 ==== Implementing your collector ==== ==== Implementing your collector ====
  
Line 528: Line 546:
 class MyCollector extends Collector class MyCollector extends Collector
 { {
- protected $idx;+  ​protected $idx;
   
- public function Prepare() +  ​public function Prepare() 
-+  
- $bResult = parent::​Prepare();​ + $bResult = parent::​Prepare();​ 
- $this->​idx = 0; + $this->​idx = 0; 
- return $bResult; + return $bResult; 
- }+  }
   
- public function Fetch() +  ​public function Fetch() 
-+  
- if ($this->​idx < 10) +     ​if ($this->​idx < 10) 
- +     ​
- $this->​idx++;​ +        $this->​idx++;​ 
- return array('​primary_key'​ => $this->​idx,​ '​name'​ => '​Server '​.$this->​idx,​ '​org_id'​ => '​Demo',​ '​description'​ => 'Test Collector'​);​ +        return array( 
- +    '​primary_key'​ => $this->​idx, ​ 
- return false; +    '​name'​ => '​Server '​.$this->​idx, ​ 
- }+    '​org_id'​ => '​Demo', ​ 
 +    '​description'​ => 'Test Collector'​ 
 +        ​); 
 +     ​
 +     ​return false; 
 +  }
 } }
  
Line 765: Line 788:
   <​brand_mapping type="​array">​   <​brand_mapping type="​array">​
     <!-- Syntax /​pattern/​replacement where:     <!-- Syntax /​pattern/​replacement where:
-      any delimiter can be used (not only /) but the delimiter cannot be present in the "​replacement"​ string+      any delimiter can be used (not only /)  
 +      ​but the delimiter cannot be present in the "​replacement"​ string
       pattern is a RegExpr pattern       pattern is a RegExpr pattern
       replacement is a sprintf string in which:       replacement is a sprintf string in which:
Line 905: Line 929:
  
 <code php> <code php>
- protected function MustProcessBeforeSynchro() +protected function MustProcessBeforeSynchro() 
-+
- // We must reprocess the CSV data obtained from the inventory script + // We must reprocess the CSV data obtained from the inventory script 
- // to lookup the Brand/Model and OSFamily/​OSVersion in iTop + // to lookup the Brand/Model and OSFamily/​OSVersion in iTop 
- return true; + return true; 
- }+}
   
- protected function InitProcessBeforeSynchro() +protected function InitProcessBeforeSynchro() 
-+
- // Retrieve the identifiers of the OSVersion since we must do a lookup based on two fields: Family + Version + // Retrieve the identifiers of the OSVersion since we must do a lookup based on two fields: Family + Version 
- // which is not supported by the iTop Data Synchro... so let's do the job of an ETL + // which is not supported by the iTop Data Synchro... so let's do the job of an ETL 
- $this->​oOSVersionLookup = new LookupTable('​SELECT OSVersion',​ array('​osfamily_id_friendlyname',​ '​name'​));​ + $this->​oOSVersionLookup = new LookupTable('​SELECT OSVersion',​ array('​osfamily_id_friendlyname',​ '​name'​));​
  
- // Retrieve the identifiers of the Model since we must do a lookup based on two fields: Brand + Model + // Retrieve the identifiers of the Model since we must do a lookup based on two fields: Brand + Model 
- // which is not supported by the iTop Data Synchro... so let's do the job of an ETL + // which is not supported by the iTop Data Synchro... so let's do the job of an ETL 
- $this->​oModelLookup = new LookupTable('​SELECT Model',​ array('​brand_id_friendlyname',​ '​name'​));​  + $this->​oModelLookup = new LookupTable('​SELECT Model',​ array('​brand_id_friendlyname',​ '​name'​));​  
- }+}
   
- protected function ProcessLineBeforeSynchro(&​$aLineData,​ $iLineIndex)+protected function ProcessLineBeforeSynchro(&​$aLineData,​ $iLineIndex
 +
 + // Process each line of the CSV 
 + if (!$this->​oOSVersionLookup->​Lookup($aLineData,​ array('​osfamily_id',​ '​osversion_id'​),​ '​osversion_id',​ $iLineIndex))
  {  {
- // Process each line of the CSV +     throw New IgnoreRowException('​Unknown OS Version'​);​
- if (!$this->​oOSVersionLookup->​Lookup($aLineData,​ array('​osfamily_id',​ '​osversion_id'​),​ '​osversion_id',​ $iLineIndex)) +
-+
-     throw New IgnoreRowException('​Unknown OS Version'​);​ +
-+
- if (!$this->​oModelLookup->​Lookup($aLineData,​ array('​brand_id',​ '​model_id'​),​ '​model_id',​ $iLineIndex)) +
-+
-     throw New IgnoreRowException('​Unknown Model'​);​ +
- }+
  }  }
 + if (!$this->​oModelLookup->​Lookup($aLineData,​ array('​brand_id',​ '​model_id'​),​ '​model_id',​ $iLineIndex))
 + {
 +     throw New IgnoreRowException('​Unknown Model'​);​
 + }
 +}
 </​code>​ </​code>​
  
extensions/itop-data-collector-base.txt · Last modified: 2019/11/12 10:27 by vdumas

";