User Tools

Site Tools


extensions:vsphere-data-collector

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
extensions:vsphere-data-collector [2019/12/16 15:44]
vdumas [Requirements]
extensions:vsphere-data-collector [2020/08/17 12:19] (current)
Line 4: Line 4:
 description_wiki : Collector for vSphere data synchronization in iTop CMDB (VM, Hypervisor, Cluster) description_wiki : Collector for vSphere data synchronization in iTop CMDB (VM, Hypervisor, Cluster)
 index_hidden ​    : yes index_hidden ​    : yes
-version ​         : 1.0.11 +version ​         : 1.0.13 
-release_dt ​      : ​2018-12-31+release_dt ​      : ​2020-07-07
 itop-version-min :  itop-version-min : 
 download_hidden ​ : http://​www.combodo.com/​itop-extensions/​vsphere-data-collector-1.0.11-413.zip download_hidden ​ : http://​www.combodo.com/​itop-extensions/​vsphere-data-collector-1.0.11-413.zip
Line 13: Line 13:
 alias-code_hidden : vsphere-data-collector alias-code_hidden : vsphere-data-collector
 alternate-name ​  : vSphere Data Collector alternate-name ​  : vSphere Data Collector
-module-lists_hidden : itop-data-collector-base/​1.0.13,​ itop-data-collector-vsphere/​1.0.11+github-module1_url ​    : https://​github.com/​Combodo/​itop-data-collector-base 
 +github-module2_url ​    : https://​github.com/​Combodo/​itop-data-collector-vsphere 
 +module-lists_hidden : itop-data-collector-base/​1.0.13,​ itop-data-collector-vsphere/​1.0.13
 diffusion ​       : iTop Hub diffusion ​       : iTop Hub
 ---- ----
Line 31: Line 33:
   * The collector can reside on any system with web access to both vSphere Web Services and iTop.   * The collector can reside on any system with web access to both vSphere Web Services and iTop.
   * Automatic creation and update of the Synchronization Data Sources in iTop.   * Automatic creation and update of the Synchronization Data Sources in iTop.
 +  * Starting with version 1.0.12 the collection mechanism is somehow extensible
  
  
 <note tip>This collector makes use of iTop's built-in Data Synchronization mechanism. For more information about how the data synchronization works, refer to [[latest:​advancedtopics:​data_synchro_overview|Data Synchronization Overview]] and relies on [[extensions:​itop-data-collector-base|Data collector Base]] mechanism</​note>​ <note tip>This collector makes use of iTop's built-in Data Synchronization mechanism. For more information about how the data synchronization works, refer to [[latest:​advancedtopics:​data_synchro_overview|Data Synchronization Overview]] and relies on [[extensions:​itop-data-collector-base|Data collector Base]] mechanism</​note>​
 +
 +
  
 ===== Revision History ===== ===== Revision History =====
 ^  Release Date  ^  Version ​ ^  Comments ​ ^ ^  Release Date  ^  Version ​ ^  Comments ​ ^
-|  ​2019-xx-xx  ​| ​ 1.0.12 | - Fix for a crash caused by a blank datastore name (thanks to David Wißen from ITOMIG for reporting it).\\ ​ - Fix for some OpenVM Tools reporting IP addresses with a trailing space (thanks to Martin Raenker from ITOMIG).| +|  ​2020-07-07  |  1.0.13 | - Differenciates error/logs between teemip NOT installed and itop REST API issue\\ - Multi configuration file\\ - New CSV collector\\ - Configurable timestamp added in the logs\\ - New option for usage: –help | 
-|  2018-12-31 ​ |  1.0.11 | Corrects regression introduced by 1.0.10 :\\ Improved support of iTop 2.4+ (obsolescence flag)\\ ​Removed a debug trace\\ ​Fix Virtual machines which "​connectionState"​ is not "​connected"​ are skipped\\ | +|  2020-04-02 ​ ​| ​ 1.0.12 | - Made the Server and Hypervisor collectors configurable\\ ​- Fix for a crash caused by a blank datastore name (thanks to David Wißen from ITOMIG for reporting it).\\ ​ - Fix for some OpenVM Tools reporting IP addresses with a trailing space (thanks to Martin Raenker from ITOMIG).| 
-|  2018-08-24 ​ |  1.0.10 | Handles TeemIp :\\ - Automatically detects if TeemIp (as an iTop module or as a standalone application) is present.\\ - Optional synchronization of IPv4 addresses\\ - Optional synchronisation of logical interfaces |+|  2018-12-31 ​ |  1.0.11 | Corrects regression introduced by 1.0.10 :\\ Improved support of iTop 2.4+ (obsolescence flag)\\ ​Removed a debug trace\\ ​Fix Virtual machines which "​connectionState"​ is not "​connected"​ are skipped\\ | 
 +|  2018-08-24 ​ |  1.0.10 | Handles TeemIp :\\ - Automatically detects if TeemIp (as an iTop module or as a standalone application) is present.\\ - Optional synchronization of IPv4 addresses\\ - Optional synchronisation of logical interfaces |
 |  2018-07-02 ​ |  1.0.9  | Robustness :\\ - more traces in debug mode (--console_log_level=9)\\ - skipping virtual machines which "​connectionState"​ is not "​connected",​ since most of their properties are not accessible and would crash PHP/Soap (with a core dump !!) | |  2018-07-02 ​ |  1.0.9  | Robustness :\\ - more traces in debug mode (--console_log_level=9)\\ - skipping virtual machines which "​connectionState"​ is not "​connected",​ since most of their properties are not accessible and would crash PHP/Soap (with a core dump !!) |
-|  2018-02-22 ​ |  1.0.8  | The ''​os_version_mapping''​ was used only for Hypervisors,​ made it work for VirtualMachines also.\\ Proper support of UTF-8 characters in regular expressions (the code now always uses the ''​u''​ modifier) |+|  2018-02-22 ​ |  1.0.8  | The ''​os_version_mapping''​ was used only for Hypervisors,​ made it work for VirtualMachines also.\\ Proper support of UTF-8 characters in regular expressions (the code now always uses the ''​u''​ modifier) |
 |  2017-04-24 ​ |  1.0.7  | Fixed the value for the status of VMs: '​production'​ instead of '​active'​. | |  2017-04-24 ​ |  1.0.7  | Fixed the value for the status of VMs: '​production'​ instead of '​active'​. |
 |  2017-03-08 ​ |  1.0.6  | - Code cleanup.\\ - Data Synchro ''​status''​ and ''​full_load_interval''​ are now homogeneous and configurable parameters for all data sources.\\ - OS Version and OS Family have their own mapping table.\\ - Fix for the Hypervisor status value (now defaults to "​production"​).\\ -  Addition of the ''​check_soap.php''​ script for troubleshooting.\\ -  Protect against a PHP crash in case of a disconnected ESX.\\ - Do not collect an IPV6 as the management IP address of a virtual machine (since the field can only contain an IPV4). Many thanks to Andrew Armstrong for proposing this fix.| |  2017-03-08 ​ |  1.0.6  | - Code cleanup.\\ - Data Synchro ''​status''​ and ''​full_load_interval''​ are now homogeneous and configurable parameters for all data sources.\\ - OS Version and OS Family have their own mapping table.\\ - Fix for the Hypervisor status value (now defaults to "​production"​).\\ -  Addition of the ''​check_soap.php''​ script for troubleshooting.\\ -  Protect against a PHP crash in case of a disconnected ESX.\\ - Do not collect an IPV6 as the management IP address of a virtual machine (since the field can only contain an IPV4). Many thanks to Andrew Armstrong for proposing this fix.|
Line 54: Line 59:
 ===== Limitations ===== ===== Limitations =====
  
 +
 +<note warning>​Data-Collector-for-VSphere is **not working** on vCenter 7.0.0</​note>​
 This version of the collector does not collect the Data Stores (they do not exist in the iTop standard data model). Logical interfaces may be collected in the case where TeemIp (module or standalone) is used. This version of the collector does not collect the Data Stores (they do not exist in the iTop standard data model). Logical interfaces may be collected in the case where TeemIp (module or standalone) is used.
  
Line 73: Line 80:
 ===== Configuration ===== ===== Configuration =====
  
-Create ​the file ''​params.local.xml''​ in the ''​conf''​ directory with the following XML content:+ 
 +{{section>​extensions:​itop-data-collector-base#​principles&​nouser&​nodate&​nofooter&​noindent&​noheader}} 
 + 
 + 
 +For example create ​the file ''​params.local.xml''​ in the ''​conf''​ directory with the following XML content:
  
 <code xml> <code xml>
Line 89: Line 100:
 </​parameters>​ </​parameters>​
 </​code>​ </​code>​
- 
  
 ^ Parameter ^ Meaning ^ Sample value ^ ^ Parameter ^ Meaning ^ Sample value ^
Line 99: Line 109:
 | vsphere_password | Password corresponding to the vSphere login | | | vsphere_password | Password corresponding to the vSphere login | |
 | vsphere_connection_options | List of PHP Stream context options to pass to the VMWarephp library in order to tune the https connection. | | | vsphere_connection_options | List of PHP Stream context options to pass to the VMWarephp library in order to tune the https connection. | |
-| contact_to_notify | The email address of an existing contact in iTop, to be notified of the results of the synchronization || 
 | default_org_id | The name of the default Organization in which the CIs (Servers, Hypervisors,​ Farms...) will be created | Demo | | default_org_id | The name of the default Organization in which the CIs (Servers, Hypervisors,​ Farms...) will be created | Demo |
- 
-==== Other optional parameters ==== 
- 
-^ Parameter ^ Meaning ^ Sample value ^ 
-| synchro_user | If the user account used for running this synchronization is //not// an Administrator,​ then its login must be specified here, since iTop allows only the administrators and the specified user to run the synchronization. | | 
- 
-==== Placeholders in the configuration of the data sources ==== 
- 
-The JSON files used to configure the data sources contain several placeholders initialized from the configuration above (''​$contact_to_notify$''​),​ but also additional placeholders specific to the data sources. These placeholders can be configured inside the ''<​json_placeholders>''​ tag in the parameters file: 
- 
-<code xml> 
-<?xml version="​1.0"​ encoding="​UTF-8"?>​ 
-  <​parameters>​ 
-    ... 
-    <​json_placeholders type="​hash">​ 
-      <​prefix>​vSphere</​prefix>​ 
-      <​full_load_interval>​60</​full_load_interval>​ 
-    </​json_placeholders> ​ 
-    ... 
-  </​parameters>​ 
-</​code>​ 
- 
-^ Parameter ^ Meaning ^ Sample value ^ 
-| full_load_interval | The delay (expressed in seconds) between two complete imports of the data. The objects which have not been detected by the collector during a timespan longer than this interval will be considered as obsolete and marked as such in iTop. Adjust this value depending on the scheduling recurrence. | 604800 | 
-| prefix | The prefix for the name of all Synchronization Data Sources in iTop. If you run several instances of the collector (to collect information from several vSphere servers), change this value so that each data source has a unique name | vSphere | 
- 
-<note tip>The file ''​params.distrib.xml''​ contains the default values for the parameters. Both files (''​params.distrib.xml''​ and ''​params.local.xml''​) use exactly the same format. But ''​params.distrib.xml''​ is considered as the reference and should remain unmodified. Should you need to change the value of a parameter, copy and modify its definition in ''​params.local.xml''​. The values in ''​params.local.xml''​ have precedence over the ones in ''​params.distrib.xml''</​note>​ 
  
 <note tip>To bypass SSL certificate validation (which is needed if your vSphere server runs with the default certificates installed by vSphere) add the folling lines in the ''​params.local.xml''​ file: <note tip>To bypass SSL certificate validation (which is needed if your vSphere server runs with the default certificates installed by vSphere) add the folling lines in the ''​params.local.xml''​ file:
Line 140: Line 122:
 </​code>​ </​code>​
 </​note>​ </​note>​
 +
 +==== Other optional parameters ====
 +
 +{{section>​extensions:​itop-data-collector-base#​optional_parameters&​nouser&​nodate&​nofooter&​noindent&​noheader}}
 +
 +
 +
  
  
Line 222: Line 211:
 <note warning>​At this stage, IPv6 are not collected. This is due to a limitation of the collector base which cannot handle such objects yet.</​note>​ <note warning>​At this stage, IPv6 are not collected. This is due to a limitation of the collector base which cannot handle such objects yet.</​note>​
  
-===== Usage =====+==== Configurable collectors ​====
  
-To launch ​the data collection ​and synchronization with iTop, run the following command (from the root directory where the data collector application is installed):+Starting with version 1.0.12 of the application, ​ //​Server// ​and //​Hypervisor//​ collectors are configurable by adding extra definitions in the XMl parameter file.
  
-<code> +This configuration contains two parts: 
-php exec.php +  - The first part (under the ''​<nowiki><​json></​nowiki>''​ tag) allows to alter the definition of the Synchro Data Source (normally fixed by the .json file associated with the collector class). This is useful to synchronise a new field which was not synchronized by default, or to change a reconciliation rule on a field, etc... Any field under the ''​attribute_list''​ part of the JSON file can be modified by specifying a new value in the configuration
-</code>+  - The second part (the ''​<source>''​ tag) defines the way to collect the data for the attribute. This tag must contain a (partial) PHP expression relative to the Hypervisor object (which is actually a [[https://​pubs.vmware.com/​vsphere-50/​topic/​com.vmware.wssdk.apiref.doc_50/​vim.HostSystem.html|HostSystem]] in the vSphere model)
  
-The following ​(optionalcommand line options are available:+Both parts of the configuration can be used independently ​(one can alter only the JSON or only the collection, or both).
  
-^ Option ^ Meaning ^ default value ^ +The example below configures the collection of the "​Serial Number"​ on Servers (from the ''​otherIdentifyingInfo[EnclosureSerialNumberTag]''​) and also uses the collected serial number as the reconciliation key between //​Hypervisors//​ and physical //Servers// (the field ''​server_id''​). 
-<nowiki>--console_log_level=<level></nowiki| Level of output to the consoleFrom -1 (none) to 9 (debug)| 6 (info) | + 
-<nowiki>--collect_only</nowiki| Run only the data collection, but do not synchronize the data with iTop | false | +<file xml config.local.xml> 
-<nowiki>--synchro_only</nowiki| Synchronizes the data previously collected (stored in the ''​data''​ directory) with iTop. Do not run the collection. | false | +<?xml version="​1.0"​ encoding="​UTF-8"?​> 
-<nowiki>--configure_only</nowiki| Check (and update if necessary) the synchronization data sources in iTop and exit. Do NOT run the collection or the synchronization | +<parameters> 
-<nowiki>--max_chunk_size=<size></nowiki| Maximum number of items to process ​in one pass, for preserving ​the memory of the systemIf there are more items to process, the application will iterate. | 1000 |+ 
 +  ​..
 + 
 +  <custom_synchro> 
 +    <​vSphereHypervisorCollector>​ 
 +      <​fields>​ 
 +        <​server_id>​ 
 +          <​source>​hardware->​systemInfo->​otherIdentifyingInfo[EnclosureSerialNumberTag]</source
 +          <json> 
 +            <​reconciliation_attcode>​serialnumber</reconciliation_attcode
 +          </json> 
 +        ​</server_id
 +      </fields> 
 +    </​vSphereHypervisorCollector>​ 
 +    <​vSphereServerCollector>​ 
 +      <​fields>​ 
 +        <​serialnumber>​ 
 +          <​source>​hardware->​systemInfo->​otherIdentifyingInfo[EnclosureSerialNumberTag]</source> 
 +        ​</serialnumber> 
 +      </​fields>​ 
 +    </​vSphereServerCollector>​ 
 +  </​custom_synchro>​ 
 +   
 +  ... 
 +   
 +</​parameters>​ 
 +</​file>​ 
 + 
 +<note important>​The special syntax with brackets ​in the ''<​source>''​ tag is currently only supported on the ''​otherIdentifyingInfo''​ property.</​note>​ 
 + 
 +===== Usage =====
  
 The execution of the command line will: The execution of the command line will:
Line 244: Line 263:
   - Upload the collected data into iTop   - Upload the collected data into iTop
   - Synchronize the collected data with the existing iTop CIs.   - Synchronize the collected data with the existing iTop CIs.
 +
 +{{section>​extensions:​itop-data-collector-base#​usage&​nouser&​nodate&​nofooter&​noindent&​noheader}}
 +
  
 ==== Troubleshooting ==== ==== Troubleshooting ====
Line 267: Line 289:
 </​code>​ If the result is not the expected one, check that your vSphere server is properly configured to run as HTTPS (which should be the default). </​code>​ If the result is not the expected one, check that your vSphere server is properly configured to run as HTTPS (which should be the default).
  
-==== Scheduling ==== 
- 
-Once you've run the data collector interactively,​ the next step is to schedule its execution so that the collection and import occurs automatically at regular intervals. 
- 
-The data collector does not provide any specific scheduling mechanism, but the simple command line ''​php exec.php''​ can be scheduled with either [[http://​en.wikipedia.org/​wiki/​Cron|cron]] (on Linux systems) or  using the [[http://​en.wikipedia.org/​wiki/​Windows_Task_Scheduler|Task Scheduler]] on Windows. 
- 
-<note tip>For optimal results, don't forget to adjust the configuration parameter ''​full_load_interval''​ in the (''​json_placeholders''​ section) to make it consistent with the frequency of the scheduling.</​note>​ 
  
 ===== Data Collection Reference ===== ===== Data Collection Reference =====
Line 279: Line 294:
 ==== Servers and Hypervisors ==== ==== Servers and Hypervisors ====
  
-In the vSphere web services SDK, Hypervisors and physical Servers are represented by the same object [[http://pubs.vmware.com/​vsphere-50/index.jsp#​com.vmware.wssdk.apiref.doc_50/​vim.HostSystem.html|HostSystem]].+In the vSphere web services SDK, Hypervisors and physical Servers are represented by the same object [[https://code.vmware.com/apis/196/vsphere/doc/​vim.HostSystem.html|HostSystem]].
  
 The information from the vSphere HostSystem object is imported in iTop into the Server object using the following mapping: The information from the vSphere HostSystem object is imported in iTop into the Server object using the following mapping:
Line 305: Line 320:
  
  
 +<note tip>The example used in the section [[#​configurable_collectors|Configurable Collectors]] shows how to collect the serial number of servers. Be aware that the information available in the property ''​otherIdentifyingInfo''​ seem to depend on the hardware manufacturer and/or the version of the hypervisor. Test on your systems before putting such a modification in production.</​note>​
 ==== Farms ==== ==== Farms ====
  
-In the vSphere web services SDK, a Farm is represented by the object [[http://pubs.vmware.com/​vsphere-50/index.jsp#​com.vmware.wssdk.apiref.doc_50/​vim.ClusterComputeResource.html|ClusterComputeResource]].+In the vSphere web services SDK, a Farm is represented by the object [[https://code.vmware.com/apis/196/vsphere/doc/​vim.ClusterComputeResource.html|ClusterComputeResource]].
  
 The information from the vSphere ClusterComputeResource object is imported in iTop into the Farm object using the following mapping: The information from the vSphere ClusterComputeResource object is imported in iTop into the Farm object using the following mapping:
Line 320: Line 335:
 ==== Virtual Machines ==== ==== Virtual Machines ====
  
-In the vSphere web services SDK, a Virtual Machine is represented by the object [[http://pubs.vmware.com/​vsphere-50/index.jsp#​com.vmware.wssdk.apiref.doc_50/​vim.VirtualMachine.html#​field_detail|VirtualMachine]].+In the vSphere web services SDK, a Virtual Machine is represented by the object [[https://code.vmware.com/apis/196/vsphere/doc/​vim.VirtualMachine.html|VirtualMachine]].
 The information from the vSphere VirtualMachine object is imported in iTop into the VirtualMachine object using the following mapping: The information from the vSphere VirtualMachine object is imported in iTop into the VirtualMachine object using the following mapping:
  
Line 357: Line 372:
 The ip_list attribute is synchronized through the dedicated synchro data source vSphere:​lnkIPInterfaceToIPAddress. The ip_list attribute is synchronized through the dedicated synchro data source vSphere:​lnkIPInterfaceToIPAddress.
 </​note>​ </​note>​
- 
-===== Collecting information from several vSphere servers ===== 
- 
-Although the current version of the collector supports only the connection to one single vSphere server, it is possible to collect data from several vSphere instances and feed the result into one iTop instance. Here is how to proceed: 
- 
-  * Copy the whole directory containing the collector application,​ one copy per vSphere server 
-  * Change the configuration file ''​conf/​local.params.xml''​ in each copy of the collector application to: 
-    - provide the proper connection information and credentials to the vSphere server 
-    - **important:​** set a unique ''​prefix''​ (in the configuration file, see [[vsphere-data-collector#​placeholders_in_the_configuration_of_the_data_sources|Placeholders above in the configuration file]]) for each vSphere server 
-  * When the collector is run, this will create (update & execute) a complete set of Data Synchronization sources for each specific vSphere server. 
-  * Schedule the execution of the different collectors at different times to avoid overloading your iTop server by concurrent data synchronizations. 
- 
-<note tip>For advanced users: actually there is no need to completely clone the whole collector application for each vSphere server. What really need to be unique for each server is: the configuration file (''​conf/​params.local.xml''​) and the ''​data''​ directory. All the other files can be symbolic links to their original version.</​note>​ 
- 
-<note tip>You may also be interested by the extension ​  ​[[itop-synchro-dashboard|Synchro Dashboard]] which provides a consolidated dashboard about the status of all the data synchronisation sources on a given iTop instance</​note>​ 
- 
- 
- 
- 
- 
- 
- 
- 
  
  
extensions/vsphere-data-collector.1576507449.txt.gz · Last modified: 2019/12/16 15:44 (external edit)

";