User Tools

Site Tools


extensions:itop-data-collector-base

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
extensions:itop-data-collector-base [2019/02/05 18:14]
vdumas [Data Collector Base]
extensions:itop-data-collector-base [2020/01/07 18:13] (current)
Line 1: Line 1:
-====== Data Collector Base ======+======= Data Collector Base ======
 ---- dataentry summary ---- ---- dataentry summary ----
 name             : Data Collector Base name             : Data Collector Base
 description_wiki : Inventory Data Collector toolkit for creating your own data collectors for iTop description_wiki : Inventory Data Collector toolkit for creating your own data collectors for iTop
 index_hidden ​    : yes index_hidden ​    : yes
-version ​         : 1.0.11 +version ​         : 1.1.2 
-release_dt ​      : ​2018-06-26 +release_dt ​      : ​2019-11-07 
-itop-version-min ​ +code             : itop-data-collector-base 
-dependencies_s ​  :  +state            : stable 
-diffusion ​       : iTop Hub, Combodo site +product_hidden ​  : not-included 
-download_url ​    : http://​www.combodo.com/​itop-extensions/​itop-data-collector-base-1.0.11-346.zip +module-lists_hidden : itop-data-collector-base/​1.1.
-module-lists_hidden : itop-data-collector-base/​1.0.11+diffusion ​       : Client Store, Combodo Site
 ---- ----
  
Line 16: Line 16:
  
  
-This module provides the base for creating an industrial data collection and synchronization application for iTop. Developers can rely on this module to perform all the heavy lifting related to the iTop data import and synchronization process in order to focus on the data collection. 
  
 +This module provides the base for creating an industrial data collection and synchronization application for iTop. Developers can rely on this module to perform all the heavy lifting related to the iTop data import and synchronization process in order to focus on the data collection.
  
 +|  [[http://​www.combodo.com/​itop-extensions/​itop-data-collector-base-1.1.2-496.zip|download]] ​ |
 ===== Features ===== ===== Features =====
  
Line 37: Line 38:
 ===== Revision History ===== ===== Revision History =====
 ^  Release Date  ^  Version ​ ^  Comments ​ ^ ^  Release Date  ^  Version ​ ^  Comments ​ ^
 +|  2019-11-07 ​ |  1.1.2  | Fix "​undefined constant TABLENAME_PATTERN"​ |
 +|  2019-10-28 ​ |  1.1.1  | Contains upgrades from both 1.0.13 and 1.1.0 \\ * Reject invalid characters for database_table_name |
 +|  2019-10-28 ​ |  1.1.0  | Based on 1.0.9 \\ * Added the specific class MySQLCollector which forces the DB connection to use UTF-8 characters |
 +|  2019-10-28 ​ |  1.0.13 ​ | * LookupTables can now be non case sensitive (since MySQL is not) \\         * Prevent a warning in SQLCollector for each "​ignored"​ attribute \\         * Improved support of iTop 2.4+ (obsolescence flag) |
 +|  2019-10-28 ​ |  1.0.12 ​ | * removed a warning in PHP 7.2 |
 |  2018-06-26 ​ |  1.0.11 ​ | Added a debug trace (visible if console_log_level=9) to show which mapping regular expression is applied (when one is applied).\\ Bug fix: properly handle utf-8 characters in the mapping table'​s regular expressions (/u modifier)\\ Make the cUrl/SSL options configurable to suit all possible combinations and security considerations. | |  2018-06-26 ​ |  1.0.11 ​ | Added a debug trace (visible if console_log_level=9) to show which mapping regular expression is applied (when one is applied).\\ Bug fix: properly handle utf-8 characters in the mapping table'​s regular expressions (/u modifier)\\ Make the cUrl/SSL options configurable to suit all possible combinations and security considerations. |
 |  2015-06-30 ​ |  1.0.10 ​ | New class of collector: ''​MySQLCollector''​ which forces the retrieved data to be encoded in UTF-8.| |  2015-06-30 ​ |  1.0.10 ​ | New class of collector: ''​MySQLCollector''​ which forces the retrieved data to be encoded in UTF-8.|
Line 68: Line 74:
 ===== Configuration ===== ===== Configuration =====
  
-At minimum, edit the file ''​params.local.xml'' ​and adjust ​the following ​values:+''​params.local.xml'' ​is the only file to edit to configure a collector. \\ 
 + 
 +At minimum ​the following ​parameters must be set in this file:
  
 <code xml> <code xml>
Line 91: Line 99:
 | stop_on_synchro_error | Whether or not to stop when an error occurs during a synchronization (''​yes''​ or ''​no''​). | no | | stop_on_synchro_error | Whether or not to stop when an error occurs during a synchronization (''​yes''​ or ''​no''​). | no |
 | console_log_level | Level of ouput to the console. From -1 (none) to 9 (debug). | 6 (info) | | console_log_level | Level of ouput to the console. From -1 (none) to 9 (debug). | 6 (info) |
 +| curl_options | When using cUrl to connect to the iTop Webservices the cUrl options can be specified in this section. The syntax is <​CURLOPT_NAME_OF_THE_OPTION1>​VALUE_1</​CURLOPT_NAME_OF_THE_OPTION1>​ where VALUE_x are either:\\ The numeric value of the option,\\ or the string representation of the corresponding PHP "​define"​ (case sensitive).\\ It is possible to define several php_curl options like in the example below| ​ |
  
 +<note tip>
 +<​code>​
 +<​curl_options>​
 +    <​CURLOPT_SSLVERSION>​CURL_SSLVERSION_SSLv3</​CURLOPT_SSLVERSION>​
 +    <​CURLOPT_SSL_VERIFYHOST>​0</​CURLOPT_SSL_VERIFYHOST>​
 +    <​CURLOPT_SSL_VERIFYPEER>​1</​CURLOPT_SSL_VERIFYPEER>​
 +  </​curl_options>​
 +</​code>​
 +</​note>​
  
 <note tip>The file ''​params.distrib.xml''​ contains the default values for the parameters. Both files (''​params.distrib.xml''​ and ''​params.local.xml''​) use exactly the same format. But ''​params.distrib.xml''​ is considered as the reference and should remain unmodified. Should you need to change the value of a parameter, copy and modify its definition in ''​params.local.xml''​. The values in ''​params.local.xml''​ have precedence over the ones in ''​params.distrib.xml''</​note>​ <note tip>The file ''​params.distrib.xml''​ contains the default values for the parameters. Both files (''​params.distrib.xml''​ and ''​params.local.xml''​) use exactly the same format. But ''​params.distrib.xml''​ is considered as the reference and should remain unmodified. Should you need to change the value of a parameter, copy and modify its definition in ''​params.local.xml''​. The values in ''​params.local.xml''​ have precedence over the ones in ''​params.distrib.xml''</​note>​
Line 127: Line 145:
  
 <code php> <code php>
-Orchestrator::​AddRequirement('​5.4.0'​);​ //​This ​code requires at least PHP 5.4 +Orchestrator::​AddRequirement('​5.4.0'​);​ //This requires at least PHP 5.4 
-Orchestrator::​AddRequirement('​1.2.0',​ '​ldap'​);​ //​This ​code requires the ldap extension, with at least version 1.2.0+Orchestrator::​AddRequirement('​1.2.0',​ '​ldap'​);​ //This requires ​at least the ldap extension version 1.2.0
 </​code>​ </​code>​
 ==== Creating the JSON definition file ==== ==== Creating the JSON definition file ====
 The simpler way to create the JSON file for a Synchro Data Source, is to export the definition of an existing data source. The simpler way to create the JSON file for a Synchro Data Source, is to export the definition of an existing data source.
   * Create the synchronisation data source in iTop, adjust its parameters (attributes,​ etc.) to suit your needs   * Create the synchronisation data source in iTop, adjust its parameters (attributes,​ etc.) to suit your needs
-  * Use the command line tool ''​dump_task.php''​ (available in the ''​toolkit''​ folder to produce the JSON file:+  * Use the command line tool ''​dump_tasks.php''​ (available in the ''​toolkit''​ folder to produce the JSON file:
  
-<​code>​php toolkit/dump_task.php --task_name="​name of the task to export"​ > collectors/​myCollector.json</​code>​+<​code>​php toolkit/dump_tasks.php --task_name="​name of the task to export"​ > collectors/​myCollector.json</​code>​
  
 Inside your data source definition you can use special placeholders to make the data source configurable by the user of the application,​ or to adjust its behavior via some special settings: Inside your data source definition you can use special placeholders to make the data source configurable by the user of the application,​ or to adjust its behavior via some special settings:
Line 152: Line 170:
 <code xml params.local.xml>​ <code xml params.local.xml>​
 <?xml version="​1.0"​ encoding="​UTF-8"?>​ <?xml version="​1.0"​ encoding="​UTF-8"?>​
-<​!-- ​ Local values for parameters. The values defined in this file have precedence over the ones defined in params.distrib.xml -->+<​!-- ​ Local values for parameters. ​--> 
 +<​!--  ​The values defined in this file have precedence over the ones defined in params.distrib.xml -->
 <​parameters>​ <​parameters>​
   <​itop_url>​https://​localhost/​trunk</​itop_url>​   <​itop_url>​https://​localhost/​trunk</​itop_url>​
Line 514: Line 533:
 </​note>​ </​note>​
  
 +<note warning>
 +If you modify the list of fields of a class already collected, you **must** update your JSON definition file, to specify what to do with the new fields. By default they are added to existing Data Synchro with ''​no update''​ and ''​no lock''​.
 +</​note>​
 ==== Implementing your collector ==== ==== Implementing your collector ====
  
Line 528: Line 550:
 class MyCollector extends Collector class MyCollector extends Collector
 { {
- protected $idx;+  ​protected $idx;
   
- public function Prepare() +  ​public function Prepare() 
-+  
- $bResult = parent::​Prepare();​ + $bResult = parent::​Prepare();​ 
- $this->​idx = 0; + $this->​idx = 0; 
- return $bResult; + return $bResult; 
- }+  }
   
- public function Fetch() +  ​public function Fetch() 
-+  
- if ($this->​idx < 10) +     ​if ($this->​idx < 10) 
- +     ​
- $this->​idx++;​ +        $this->​idx++;​ 
- return array('​primary_key'​ => $this->​idx,​ '​name'​ => '​Server '​.$this->​idx,​ '​org_id'​ => '​Demo',​ '​description'​ => 'Test Collector'​);​ +        return array( 
- +    '​primary_key'​ => $this->​idx, ​ 
- return false; +    '​name'​ => '​Server '​.$this->​idx, ​ 
- }+    '​org_id'​ => '​Demo', ​ 
 +    '​description'​ => 'Test Collector'​ 
 +        ​); 
 +     ​
 +     ​return false; 
 +  }
 } }
  
Line 765: Line 792:
   <​brand_mapping type="​array">​   <​brand_mapping type="​array">​
     <!-- Syntax /​pattern/​replacement where:     <!-- Syntax /​pattern/​replacement where:
-      any delimiter can be used (not only /) but the delimiter cannot be present in the "​replacement"​ string+      any delimiter can be used (not only /)  
 +      ​but the delimiter cannot be present in the "​replacement"​ string
       pattern is a RegExpr pattern       pattern is a RegExpr pattern
       replacement is a sprintf string in which:       replacement is a sprintf string in which:
Line 905: Line 933:
  
 <code php> <code php>
- protected function MustProcessBeforeSynchro() +protected function MustProcessBeforeSynchro() 
-+
- // We must reprocess the CSV data obtained from the inventory script + // We must reprocess the CSV data obtained from the inventory script 
- // to lookup the Brand/Model and OSFamily/​OSVersion in iTop + // to lookup the Brand/Model and OSFamily/​OSVersion in iTop 
- return true; + return true; 
- }+}
   
- protected function InitProcessBeforeSynchro() +protected function InitProcessBeforeSynchro() 
-+
- // Retrieve the identifiers of the OSVersion since we must do a lookup based on two fields: Family + Version + // Retrieve the identifiers of the OSVersion since we must do a lookup based on two fields: Family + Version 
- // which is not supported by the iTop Data Synchro... so let's do the job of an ETL + // which is not supported by the iTop Data Synchro... so let's do the job of an ETL 
- $this->​oOSVersionLookup = new LookupTable('​SELECT OSVersion',​ array('​osfamily_id_friendlyname',​ '​name'​));​ + $this->​oOSVersionLookup = new LookupTable('​SELECT OSVersion',​ array('​osfamily_id_friendlyname',​ '​name'​));​
  
- // Retrieve the identifiers of the Model since we must do a lookup based on two fields: Brand + Model + // Retrieve the identifiers of the Model since we must do a lookup based on two fields: Brand + Model 
- // which is not supported by the iTop Data Synchro... so let's do the job of an ETL + // which is not supported by the iTop Data Synchro... so let's do the job of an ETL 
- $this->​oModelLookup = new LookupTable('​SELECT Model',​ array('​brand_id_friendlyname',​ '​name'​));​  + $this->​oModelLookup = new LookupTable('​SELECT Model',​ array('​brand_id_friendlyname',​ '​name'​));​  
- }+}
   
- protected function ProcessLineBeforeSynchro(&​$aLineData,​ $iLineIndex)+protected function ProcessLineBeforeSynchro(&​$aLineData,​ $iLineIndex
 +
 + // Process each line of the CSV 
 + if (!$this->​oOSVersionLookup->​Lookup($aLineData,​ array('​osfamily_id',​ '​osversion_id'​),​ '​osversion_id',​ $iLineIndex))
  {  {
- // Process each line of the CSV +     throw New IgnoreRowException('​Unknown OS Version'​);​
- if (!$this->​oOSVersionLookup->​Lookup($aLineData,​ array('​osfamily_id',​ '​osversion_id'​),​ '​osversion_id',​ $iLineIndex)) +
-+
-     throw New IgnoreRowException('​Unknown OS Version'​);​ +
-+
- if (!$this->​oModelLookup->​Lookup($aLineData,​ array('​brand_id',​ '​model_id'​),​ '​model_id',​ $iLineIndex)) +
-+
-     throw New IgnoreRowException('​Unknown Model'​);​ +
- }+
  }  }
 + if (!$this->​oModelLookup->​Lookup($aLineData,​ array('​brand_id',​ '​model_id'​),​ '​model_id',​ $iLineIndex))
 + {
 +     throw New IgnoreRowException('​Unknown Model'​);​
 + }
 +}
 </​code>​ </​code>​
  
extensions/itop-data-collector-base.1549386865.txt.gz · Last modified: 2019/02/05 18:14 (external edit)

";