IT38356: DATA COLLECTOR UPGRADE FAILS DUE TO CACHED DATA UPLOAD FAILURE

APAR status

• Closed as program error.

Error description

• Data collector upgrades have been observed to fail due to a
  failure to upload cached performance data samples. Evidence of
  this can be seen in the traceMessage_x.log files, for example:
  [StoreAndForward_CacheUploader]
  [UploadUtil.uploadPerformanceMonitorResult] [WARNING] summary
  file not created?
  
  This is what prevented the data collector upgrade from
  finishing.
  We can see that the upgrade installer will start to extract and
  replace the data collector, but then we see the following:
  2021-09-08 09:00:32.689+0200 [pool-1-thread-2020]
  [DownloadTask.checkAndWaitForCacheUpload] [WARNING] Completed DC
  upgrade download, but waiting for cache upload to complete.
  2021-09-08 09:21:11.001+0200 [StoreAndForward_CacheUploader]
  [CacheUploader.run] [INFO] Store And Forward Thread waiting for
  1m since upload failed
  2021-09-08 09:21:18.017+0200 [Thread-10168]
  [CollectorUtil.sendResultToDeviceServer1] [WARNING] Appending
  files to cache as cache upload is running: E:\IBMSW\Cloud_DataC
  ollector_windows/log/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx\perf_
  monitor\12345
  2021-09-08 09:21:32.912+0200 [pool-1-thread-2020]
  [StoreAndForwardCache.cacheUploadRunning] [INFO] Uploader Status
  false
  2021-09-08 09:21:44.887+0200 [Respond]
  [CollectorUtil.handleResult] [INFO] Removing from queue batched
  process ID -14128
  2021-09-08 09:22:11.011+0200 [StoreAndForward_CacheUploader]
  [CacheUploader.run] [ERROR] Error sending entry from cache,
  skipping cache depletion for now
  2021-09-08 09:22:12.098+0200 [StoreAndForward_CacheUploader]
  [StoreAndForwardCache.getOldestCacheEntry] [INFO] Files to be
  uploaded [E:\IBMSW\Cloud_DataCollector_windows\cache\xxxxxxxx-x
  xxx-xxxx-xxxx-xxxxxxxxxxxx\xxxxx\12345.zip,E:\IBMSW\Cloud_DataCo
  xxx-xxxxxxxxxxxx\12345\PENDING_DELETION]
  2021-09-08 09:22:12.114+0200 [StoreAndForward_CacheUploader]
  [UploadUtil.uploadPerformanceMonitorResult] [WARNING] summary
  file not created
  2021-09-08 09:22:12.114+0200 [StoreAndForward_CacheUploader]
  [CacheUploader.run] [INFO] Store And Forward Thread waiting for
  1m since upload failed
  ____________________________________________________
  The defect is against component: 5608TPCSI
  Server/Manager build/release: 5.4.x
  Problem as described by customer:  Data collector upgrade fails
  Initial customer impact: low
  

Local fix

• The workaround is to manually delete the content of the
  referenced cache folder (in this example, ?12345? as shown in
  the logs). Upon deletion of this folder, the data collector
  upgrade should resume and complete.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Storage Insights users upgrading Data Collectors         *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * Storage Insights Data Collector upgrades have been observed  *
  * to fail due to a failure to upload cached performance data   *
  * samples. Evidence of this can be seen in the                 *
  * traceMessage_x.log files.                                    *
  *                                                              *
  * This can interrupt and prevented the data collector upgrade  *
  * from finishing.                                              *
  *                                                              *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  

Problem conclusion

• The fix for this APAR is targeted for the following release:
  
  IBM Storage Insights 4Q21   [ 54X-IBM-SI ]
  
  ( release target 4Q 2021 / November )
  
  The target dates for future releases do not represent a formal
  commitment by IBM. The dates are subject to change without
  notice.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  STORAGE INSIGHT

• Fixed component ID

  5608TPCSI

Applicable component levels