IT38041: SPECTRUM VIRTUALIZE PROBES FAIL; PROBE DATA PROCESSING TAKES LONGER THAN ALLOTTED AMOUNT OF TIME. TASK FAILED OR EXPIRED

APAR status

• Closed as program error.

Error description

• Storage Insights probes against storage devices may
  experience a timeout due to the probe data processing taking
  longer than the allotted time for the probe to complete.
  
  In these situations, the Storage Insights data collector
  successfully gathers all probe data from the device and uploads
  it to Storage Insights; Storage Insights then takes too long to
  process the received data, and thus the probe is marked as
  failed or task expired.
  
  Internally this is logged as follows:
  
  Error extracting properties form TimeSeriesDataQueryClient for
  tenant: <tenant_uuid>uuid: <device_uuid>
  com.ibm.srm.timeseries.query.api.TimeSeriesQueryException: Error
  extracting properties form TimeSeriesDataQueryClient for tenant:
  <tenant_uuid>uuid: <device_uuid> at com.ibm.srm.dataprocessor.c
  ommon.util.ProbeUtil.getPropertiesComponent(ProbeUtil.java:1933)
  edProperties(ProbeUtil.java:2094)at com.ibm.srm.dataprocessor.co
  Processor.fillComponentUserDefinedProperties(AbstractProbeDataP
  rocessor.java:303)at com.ibm.srm.dataprocessor.common.probe.api.
  Processor.getComponentUserDefinedProperties(AbstractProbeDataPr
  ocessor.java:285)at com.ibm.srm.dataprocessor.common.probe.api.A
  Processor.processProbeData(AbstractProbeDataProcessor.java:199)a
  Processor.processKafkaProbeData(AbstractProbeDataProcessor.java
  :2042)at com.ibm.srm.dataprocessor.probe.service.impl.AbstractDa
  cessorService.processOnMessage(AbstractDataProcessorService.jav
  a:461)at com.ibm.srm.dataprocessor.probe.service.impl.ProbeDataP
  sorService.onMessage(ProbeDataProcessorService.java:214)at com.i
  un$1(ConsumerTask.java:237)at com.ibm.srm.utils.kafka.consumer.t
  259/0x00000000b00a8490.run(UnknownSource) at java.util.concurren
  readPoolExecutor.java:1160)at java.util.concurrent.ThreadPoolExe
  lExecutor.java:635)at java.lang.Thread.run(Thread.java:822) Caus
  com.ibm.srm.timeseries.query.api.TimeSeriesQueryException:
  getComponentsConfigDataBySystem failed with exception
  java.net.SocketTimeoutException: SocketTimeoutException invoking
  https://time-series-data.si.svc.cluster.local:9443/srm/tsdqueri
  es/v1/config/tenants/<tenant_uuid>/storage_systems/<device_uuid
  >/io_groups:connect timed out at com.ibm.srm.timeseries.query.ap
  sDataQueryClient.getComponentsConfigDataBySystem(TimeSeriesData
  QueryClient.java:282)at com.ibm.srm.dataprocessor.common.util.Pr
  sComponent(ProbeUtil.java:1921)... 12 more
  
  ____________________________________________________
  The defect is against component:
  5608TPCSI
  
  Server/Manager build/release:
  5.4.x
  
  Problem as described by customer:
  Storage Virtualize probe failure
  
  Initial customer impact (low/med/high):
  med
  

Local fix

• To increase the probe timeout a new property can be
  added to:
  
  <DC_INSTALL>\conf\setup.properties
  
  probeTimeout =  <value_in_ms>;
  
  For example, to set 120 minutes in ms:
  
  probeTimeout = 7200000;
  
  
  Following the change a Data Collector restart is needed.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Storage Insights users monitoring Spectrum Virtualize    *
  * devices                                                      *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * Storage Insights probes against storage devices may          *
  * experience a timeout due to the probe data processing        *
  * taking longer than the allotted time for the probe to        *
  * complete.                                                    *
  *                                                              *
  * In these situations, the Storage Insights data collector     *
  * successfully gathers all probe data from the device and      *
  * uploads it to Storage Insights; Storage Insights then takes  *
  * too long to process the received data, and thus the probe is *
  *                                                              *
  * marked as failed or task expired.                            *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  

Problem conclusion

• The fix for this APAR is targeted for the following release:
  
  IBM Storage Insights 1Q22   [ 54X-IBM-SI ]
  ( release target 1Q 2022 / February )
  
  The target dates for future releases do not represent a formal
  commitment by IBM. The dates are subject to change without
  notice.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  STORAGE INSIGHT

• Fixed component ID

  5608TPCSI

Applicable component levels