IT40655: ALL SITES VIRTUAL MACHINE BACKUP JOBS ARE STUCK IF ONE VSNAP SERVER DOES NOT RESPOND OR IS SLOW

APAR status

• Closed as program error.

Error description

• All sites virtual machine backup jobs are stuck. The job log can
  stop at following entry:
  
  
  CTGGA0068,Discovery completed
  CTGGA0506,Get vadp backup info
  
  
  By checking virgo log, the next step after "Get vadp backup
  info" is "Getting all Storage servers for job". And this step
  returns error:
  
  
  [date-time] INFO  asyncTaskExecutor-6
  c.s.d.x.s.protection.common.StorageServerLoaderImpl
   1648251000030 Getting all Storage servers for job 1001
  ...
  [date-time] INFO  asyncTaskExecutor-6
  c.c.e.s.common.storage.vsnap.VsnapStorageConnection
   1648251000030 Vsnap Call
  https://<;vsnap-server>:8900/api/system method GET
  ...
  [date-time] ERROR pool-36-thread-17
  c.c.e.s.c.s.factory.updater.AsynchronousStorageStateUpdater
    Failed to update statistics for storage with ID 2001 com.cata
  logic.ecx.serviceprovider.common.storage.management.exception.S
  torageManagementException:Storage management get pool exception.
  pools on server <vsnap-server>Storage <vsnap-server>:8900 is in
  Invalid state , status code :503 reason : 503 SERVICE
  UNAVAILABLE
                 at com.catalogic.ecx.serviceprovider.common.stor
  age.vsnap.VsnapStorageServer.getManagementPools(VsnapStorageSer
  ver.java:2309)
  ...
  
  The vSnap call to get <vsnap-server> returns one error and is
  slow because this vSnap server does not respond. However, the
  backup jobs on different sites other than the one where this
  problematic vSnap server locates are also affected. At the
  beginning of each backup, IBM Spectrum Protect Plus tries to
  load all vSnap servers, including querying each one. This is not
  necessary and can become a problem when one of the vSnap is not
  responding or very slow.
  
  
  
  Versions affected: 10.1.*
  

Local fix

• Reboot the problematic vSnap server
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.9 and 10.1.10           *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply the fixing level when available. This problem is       *
  * currently projected to be fixed in IBM Spectrum Protect Plus *
  * level 10.1.11. Note that this is subject to change at the    *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• The improvement for this issue is to only try to load the vSnap
  servers for the target site of the job.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• Hypervis
  

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels