Learn more about installing and administering data collectors

To help you install data collectors, information is provided about extracting the files, connecting securely to IBM® Storage Insights Pro, and installing the data collector service. Guidelines are provided for setting the properties of your data collectors so that they continue to collect performance metadata during service outages.

Extract the data collector

The files that are required to install and run the data collector are provided in a compressed file. Download the compressed file on the operating system where you want to run the data collector. When the download is complete, use an extraction utility that is provided with the operating system to extract the file contents to a directory of your choice. Ensure that the directory that you select is empty and that the directory name and the names of any directories in the directory path follow these naming conventions:

Directory names must not include special characters such as these:
```
% & ! ( ) { } [ ] ; , ' ` ^ = $ # ~ + @ \ / : * ? " < > |
```
If you plan to add any non-IBM storage systems, directory names must not start with any of the Windows reserved words such as these:
```
CON, PRN, AUX, NUL, COM1, COM2, COM3, COM4, COM5, COM6, COM7, COM8, COM9,
LPT1, LPT2, LPT3, LPT4, LPT5, LPT6, LPT7, LPT8, LPT9
```
The total characters for the names in the directory path must not exceed 256 characters.

Important:

Where the data collector is installed on a Windows system, the total characters for the names in the directory path should not exceed 50 characters.
On the server or virtual machine where you install the data collector, you must provide at least 1 GB of RAM and 3 GB of disk space.

Ensure that data collectors can access IBM Storage Insights

Data collectors send metadata about your devices to the endpoint for IBM Storage Insights at https://insights.ibm.com. To view that metadata and monitor your devices in IBM Storage Insights, you must ensure that your data collectors can access that endpoint URL.

To verify that a data collector can connect to https://insights.ibm.com, run the following command on the server or virtual machine where it's installed:

curl https://insights.ibm.com -I

If the connection is successful, output similar to the following is displayed:

HTTP/2 200
date: Fri, 23 Jul 2021 01:30:30 GMT
content-type: text/html
cf-ray: 6733ed73ff281d39-BLR
last-modified: Thu, 17 Sep 2020 12:13:01 GMT
vary: Accept-Encoding
cf-cache-status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
x-amz-request-id: 2a462ec5-c60d-4abb-bff9-8017d7b216fc
x-clv-request-id: 2a462ec5-c60d-4abb-bff9-8017d7b216fc
x-clv-s3-version: 2.5
server: cloudflare

A Secure Sockets Layer (SSL) certificate is used to secure communication between the data collector and IBM Storage Insights.

Firewall requirements: To connect to your instance of IBM Storage Insights, your firewall must be configured to allow outbound communication on the default HTTPS port 443 using the Transmission Control Protocol (TCP). The User Datagram Protocol (UDP) is not supported.

Manage disk space during service outages

The data collectors continue to collect performance metadata if there is a disruption in communication between IBM Storage Insights and the data collectors. To avoid gaps in your performance charts and reports, the performance metadata that was collected during the outage is loaded into IBM Storage Insights when the connection is reestablished.

The data collectors require a certain amount of disk space to store the performance history metadata if there is a communication disruption.

The following properties are available to manage the disk space requirements on the server or virtual machine where the data collector is installed.

Table 1. Disk space requirements for data collectors
Property	Description	Default value (MiB)
`cacheMinFreeSpaceMiB`	The minimum amount of space that must be available on the server when performance metadata is collected. For example, if the property value is set to 2048 MiB, the data collector stops collecting metadata when there is only 2048 MiB remaining on the server.	2048
`cacheMaxTotalSpaceMiB`	The maximum amount of disk space that the data collector can use to store performance data during a service outage. Rule of thumb: 2 MiB of space is recommended for every 1000 volumes that you're monitoring. If extended periods of network outages are being experienced you need to increase this setting to retain all the data collected during network outages.	240

Important: Ensure that the server or virtual machine where the data collector is installed has the following available disk space, as a minimum:

cacheMinFreeSpaceMiB + cacheMaxTotalSpaceMiB

You can set the properties in the custom.properties file in the <data_collector_install_directory>/conf directory.

Configure how often connection status is updated for candidate devices

The connections between your data collectors and the devices that are visible to them are tested automatically by IBM Storage Insights. This test can help you track when the device was last contacted and if a device can’t be reached.

For devices that are monitored by the data collector, the connection status is checked frequently, such as when probes and performance monitors collect metadata, credentials are changed, connections are tested manually, and other operations. If a monitored device is unreachable, metadata can’t be collected.

For devices that are not monitored but are candidates to be monitored, the connection between a device and the data collector is tested every 10 minutes. If a candidate device is unreachable, the data collector can’t be assigned to monitor it.

To change frequency of how often the connection between a candidate device and the data collector is tested, you can configure the connectivityCheckIntervalMin property in the custom.properties file for the data collector. The custom.properties file is located in the <data_collector_install_directory>/conf directory on the server or virtual machine where the data collector is installed.

Run the data collector service installer script

A service is used to run the data collector. To install this service, run the installDataCollectorService.bat script for Windows operating systems or the installDataCollectorService.sh script for AIX® and Linux® operating systems.

To run the script, you must have Administrator authority on Windows operating systems or root authority on AIX or Linux operating systems.

After the data collector service starts, the data collector runs continuously unless you stop or uninstall the service.

Antivirus software considerations

Installing a data collector involves making use of your operating system in manners typical for installing new application software. If your antivirus software is set on the maximum mode, it might prevent some of your changes from being accepted.

To verify that your deployment completes correctly, enable your antivirus software product on the target server or virtual machine to allow new executable files to be created in the installation directory of the data collector. If your antivirus software prevents the monitoring of some of your devices, such as XIV®, Hitachi VSP, Dell EMC, NetApp, or Pure storage, add the related executable files for those devices to your allowlist.

McAfee tip for Windows:

If McAfee Adaptive Threat Protection is enabled on the server where an IBM Storage Insights data collector is installed, it might prevent the data collector from starting or stopping. To avoid this issue, open McAfee Adaptive Threat Protection and go to settings. In the Real Protect Scanning (Windows only) section, verify whether Enable client-based scanning is selected. If so, select Low from the Sensitivity level list.
If Symantec Endpoint Protection is enabled on the server where an IBM Storage Insights data collector is installed, it prevents the data collector from running properly. To avoid this issue, remove or disable Symantec Endpoint Protection on the server where the data collector is installed.
If Trellix endpoint security is enabled on the server where an IBM Storage Insights data collector is installed, it may prevent the data collector from running properly. To avoid this issue, remove or disable Trellix endpoint security on the server where the data collector is installed.
If Forcepoint One Endpoint is enabled on the server where an IBM Storage Insights data collector is installed, it may prevent the data collector from running properly. To avoid this issue, remove or disable Forcepoint One Endpoint on the server where the data collector is installed.

For more information about McAfee Adaptive Threat Protection, see the following links:

Frequently asked questions about data collectors

View answers to common questions about installing and using data collectors.

Do I need a data collector if I'm using IBM FlashSystem® devices?

Usually, IBM Storage FlashSystem devices do not require a data collector. However, if you need to add switches, hosts, and legacy systems, a data collector is essential. For more information about downloading and installing data collectors, see Downloading and installing data collectors.

Do I need to prepare my environment before getting started with IBM Storage Insights?

Because IBM Storage Insights is an IBM Cloud® service, getting started is a snap. However, every environment is different, so to ensure that your getting started experience goes smoothly, see this handy checklist for some hints and tips: Before you begin checklist for IBM Storage Insights.

What operating systems are supported for the data collector?

You can install the data collector on the following operating systems:

Windows Server 2016 and later.
The Linux data collector runs on 64-bit Linux operating systems on x86-64 and PPC64LE systems only. The supported Linux operating systems for x86-64 are Red Hat® Enterprise Linux 7 or later and CentOS 7 or later.
The supported Linux operating system for PPC64LE is Red Hat Enterprise Linux 7.x on POWER8®® and Red Hat Enterprise Linux 8.x. on POWER10®. The data collector on Linux PPC64LE has the additional limitation that you cannot monitor IBM FlashSystem A9000, XIV, IBM Storage Accelerate, and non-IBM devices.
Restriction: The data collector is not supported for Linux operating systems on IBM Power® systems (PPC64, Big Endian).
POWER6® or later systems that use AIX 7.x or later. The AIX data collector can run on a physical AIX installation or a logical partition (LPAR).

Tip: To avoid potential problems, ensure that the operating system on the server or virtual machine where you install the data collector has general or extended support for maintenance and security.

Important: IBM support for AIX 7.1 on POWER6 systems ends in April 2023. At that point, IBM Storage Insights will no longer support deploying data collectors to those platforms. It's strongly recommended that you deploy your data collectors to supported versions of AIX and POWER®® systems. For more information about end of support for AIX versions, AIX support lifecycle information.

What is involved in downloading data collectors?

Data collectors are downloaded as compressed files that can be extracted and installed on any server that runs Windows, Linux, or AIX operating systems or virtual machines that have access to both the internal SAN attached infrastructure as well as access to the internet to communicate with the cloud portal.

See Downloading and installing data collectors.

What compute, network, and memory resources do data collectors use?

The data collector is a light-weight application. On the server or virtual machine where you install the data collector, you must provide at least 1 GB of RAM and 3 GB of disk space.

Learn more about how disk space is used during service outages.

How can I configure the data collector to use a proxy server?

You can specify whether you want to connect to a proxy server when you install the data collector. You can also add the user's credentials to authenticate with the proxy server.

See Adding or changing the connection to the proxy server.

What do I need to configure on my firewall to enable the data collector to access the IBM cloud?

If you do not have a proxy server, then ensure that the default HTTPS port 443 is open on the firewall. You should not need to configure it for the FQDN.

Tip: If a firewall exists between the data collector and the storage systems that it monitors, configure the firewall to allow SNMP traffic.

Can I get a new data collector if something happens to my current data collector?

If the data collector is deleted or otherwise stops operating, IBM Storage Insights Pro will be unable to gather any new information. Information that has already been collected by IBM Storage Insights Pro will be kept. To restart the data collection, download a data collector and reinstall it.

What data does IBM Storage Insights collect? How is it used?

IBM Storage Insights collects two types of data: metadata for devices and diagnostic data for support logs.

Metadata for devices

The following metadata about your devices is collected regularly and streamed to IBM Storage Insights:

Information about the status of a device and its internal resources.
Information about the configuration of a device, such as name, firmware, and capacity.
Information about the internal resources of a device, such as volumes, pools, nodes, ports, and disks. This information includes the names and the configuration and capacity metrics for each internal resource.
Information about the performance of devices and internal resources such as pools, volumes, and ports.

This metadata is never stored locally, and is encrypted with 128-bit encryption while streaming and 256-bit encryption while at rest. For more information about the metadata that is collected and how it's used, check out the External link icon

IBM Storage Insights Security Guide.

Diagnostic data for support logs

Support logs for your IBM block storage systems are only collected when a support ticket is open and you request it. The logs include diagnostic information so that IBM Support can more efficiently and quickly investigate and close tickets. In Blue Diamond environments, support logs are sent to a Blue Diamond Enhanced Secure Support environment, just as if you were to do it manually.

Important: IBM Storage Insights can't access the actual application data that is stored on your storage systems.

When is metadata and diagnostic data collected?

IBM Storage Insights collects metadata about the configuration, status, capacity, and performance of your monitored devices. Diagnostic data is also collected into log packages and added to support tickets. The following types of tasks are used to collect that information:

Probes

Probes collect status, configuration, and capacity metadata once every 24 hours.

For some devices, additional probes are automatically run when specific events occur on those devices, or when many events are detected in a short time period. To avoid performance bottlenecks in those cases, probes are not run more than once every 20 minutes.

Performance Monitors

Performance Monitors collect performance metadata every 5 minutes.

For Dell EMC storage systems that manage block storage, performance metadata is automatically collected every fifteen minutes.
For IBM Spectrum® Scale storage systems that are enabled to collect performance metadata for file storage, the collection schedule can be configured when or after the storage system is added for monitoring. The intervals that can be selected for collecting performance metadata are every 5, 10, 15, 20, 30 or 60 minutes.

Tips:

After probes and performance monitor run, the GUI is updated automatically to display the latest metadata for your monitored devices.
Only metadata about your monitored devices is collected. The actual data that is stored on devices is never accessed or collected.

How will I know if a data collector stops working?

You will get an email and a message is shown at the bottom of the page of the GUI. To check the status of the data collector, you can click Configuration > Data Collectors.

Learn more Resolving connection issues.

How do I uninstall the data collector from a server and install it on a different server?

To uninstall the data collector, log in as Administrator on Windows operating systems or as a user with root privileges on AIX or Linux operating systems. After you uninstall, remove the data collector entry in the GUI. You must log in to the GUI as Administrator.

To install the data collector, choose an operating system, copy or download the data collector to the server where you want to install it, extract the data collector, and then add the storage systems that you want to monitor.

See Removing and uninstalling data collectors and Downloading and installing data collectors.

When I add a storage system, I provide user credentials and an IP address for that storage system so IBM Storage Insights can collect metadata about it. Is that information stored onsite at my location or is it storage in the IBM Cloud?

The user credentials and IP addresses that you provide for connecting to a storage system are encrypted and stored securely in the IBM Cloud instance for IBM Storage Insights. To collect metadata for the storage system, that information is transmitted over a secure channel to the locally installed data collector. The information is never stored on the host where the data collector is installed; it exists temporarily in the application memory for the data collector only.

Do the user credentials (user name and password) that I provide for connecting to a storage system have any special requirements?

Yes. The role or user group that is assigned to the user name must have the appropriate privileges to collect configuration, capacity, and performance metadata about the storage system. The type of storage system and the metadata that you want to collect determine the privileges that are required.

Learn more User roles for collecting metadata.

How often can I collect performance metadata for the storage resources that I monitor with IBM Storage Insights Pro?

The performance metadata for IBM storage systems that manage block storage is automatically collected every five minutes and the performance metadata for Dell EMC storage systems that manage block storage is automatically collected every fifteen minutes.

For IBM Storage Scale storage systems that are enabled to collect performance metadata for file storage, the collection schedule can be configured when or after the storage system is added. The intervals that can be selected for collecting performance metadata are every 5, 10, 15, 20, 30 or 60 minutes.

Do I need to back up my metadata on-premises in case a restore is required?

No, and it's not necessary. A backup of all IBM Storage Insights metadata is automatically stored in the ISO-certified IBM cloud in the unlikely event that a restore is required.

Can I create different views of the dashboard?

Yes. If you have the Administrator role in IBM Storage Insights, you can create dashboards to selectively monitor specific storage systems in your environment. For example, you might want a dashboard for each of your data centers that monitors all of the storage systems in the data center and another dashboard for your production systems.

If you don't have an Administrator role, contact a person within your organization who has that role to help create a custom dashboard for you.

See Creating customized dashboards to monitor your storage and Adding and removing users.

How can I add a DS8000® storage system that uses SSL 3 or earlier versions or that use MD5 signed certificates?

See Adding DS8000 storage systems that use SSLv3 and MD5 signed certificates.

Are there any special characters that are restricted?

When you define custom tags, specify alert policy names, customize device names, and enter other configurable text, avoid using the following special characters: $ ~ ` \ / [ ] | ; : ' " < >.