IT31765: CTGGA2025 - VM BACKUP JOB HUNG - PROXY IS NOT REDUCING CORRECTLY THE NUMBER OF AVAILABLE DATA STREAMS

APAR status

• Closed as program error.

Error description

• Spectrum Protect Plus VM backup jobs are hanging and will not be
  further processed.
  This is because the number of data streams can be incorrectly
  calculated (see number of currently processing data streams in
  below example). This can happen after errors causing orphaned
  data streams not being cleaned up.
  Example of processing data streams shown at the joblog:
  INFO,...,CTGGA2025,Job progress: VMs Queued(0) Active(2)
  Completed(142)
  WARN,...,CTGGA0682,VM XX-VM1 proxy reservation set to site
  YY-Site and available proxies are currently processing 33 data
  streams with a soft cap limit of 32  retrying in 30 seconds.
  .
  From that time on - these messages will show up at the joblog
  without any alterations.
  While the joblog indicates that data streams still will be
  processed, the VIRGO log shows NO running tasks:
  INFO ... 1579024800137 Found 2 enabled VADP proxies for site
  YY-Site
  INFO ... 1579024800137 vadp
  ::https://ww.xx1.yy1.ww1:8098/system/workinfo ::200
  ::{"RunningTasks": 0,  "AvailableMemory": 37193284,
  "AvailableCores": 24,  "maxTasks": 0}
  INFO ... 1579024800137 vadp
  ::https://ww.xx2.yy2.ww2:8098/system/workinfo ::200
  ::{"RunningTasks": 0,  "AvailableMemory": 29295240,
  "AvailableCores": 8,  "maxTasks": 0}
  INFO ... 1579024800137 skipping busy proxy
  https://ww.xx1.yy1.ww1:8098
  INFO ... 1579024800137 skipping busy proxy
  https://ww.xx2.yy2.ww2:8098
  
  
  IBM Spectrum Protect Versions Affected:
  IBM Spectrum Protect Plus 10.1.x
  
  
  Initial Impact: Medium
  
  Additional Keywords: SPP, SPPLUS, TS003231282, hang, wait,
  stalled
  

Local fix

• As a temporary work around for the VM backup hung - please
  restart VIRGO by using following command from the Spectrum
  Protect Plus 'serveradmin' console:
     systemctl restart virgo
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.5                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed IBM Spectrum Protect Plus levels       *
  * 10.1.5 patch1 and 10.1.6. Note that this is subject to       *
  * change at the discretion of IBM.                             *
  ****************************************************************
  

Problem conclusion

• This problem has been fixed so that after encountering condition
  that job hung due to proxy incorrectly calculated the number of
  data streams, the new behavior is that the job will properly
  proceed as expected.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels