IT45517: BACKUP TO AZURE CLOUD FAILURE WITH CTGGA0309 DUE TO TIMING OUT

APAR status

• Closed as program error.

Error description

• Backup to AZURE using a bandwidth of less than 5 MB/s have
  frequent copy to cloud failures.
  
  
  JOBLOG shows this error:
  ERROR,..,<<date>>,2,CTGGA0309,Copy failed for snapshot (ID:
  1419) from source [server: xxx  volume: spp_1028_xxx  snapshot:
  spp_1028_2365_xxx] to target [server:
  https://core.windows.net:443  volume: xxx]. Error:
  TransferError: Transfer failed: The data transfer was cancelled
  because the upload of data to the target cloud server or
  repository server did not make progress. Ensure that the vSnap
  server has adequate connectivity to the cloud server or
  repository server.
  
  Currently, the default concurrency preferences for cloud offload
  start to break down with low throttle values (5 MB/s or less).
  
  A cloud offload job with a low throttle value will fail.
  
  IBM Spectrum Protect Plus Versions Affected:
  IBM Spectrum Protect Plus 10.1.3.0 and later
  
  
  Additional Keywords: SPP, SPPlus, TS012548138, Storage Protect
  Plus, AZURE, throttle
  

Local fix

• For vSnaps with a low bandwidth throttle (5 MB/s) where cloud
  offload jobs are timing out, the following commands can be run
  on the vSnap to reduce the concurrency of the vSnap to ensure
  that the timeouts no longer occur:
  
  vsnap system pref set --name cloudOffloadThreads --value 1
  vsnap system pref set --name cloudMaxStreams --value 1
  vsnap system pref set --name cloudStallCount --value 60
  vsnap system pref set --name cloudStallInterval --value 60
  
  Note that setting these preferences may cause a job to take
  longer, but the job will no longer timeout.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.3 till 10.1.16          *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply the fixing level when available. This problem is       *
  * projected to be fixed in IBM Spectrum Protect Plus level     *
  * 10.1.17. This is subject to change at the discretion of IBM. *
  ****************************************************************
  

Problem conclusion

• Before IBM Spectrum Protect Plus 10.1.17, cloud offloads with a
  source vSnap configured with a low bandwidth throttle (such as 1
  MB/s or 2 MB/s) time out (causing the job to fail). For low
  bandwidth throttles, the default level of concurrency on the
  vSnap would cause requests from the vSnap to the cloud endpoint
  to starve one another, possibly resulting in a timeout. In IBM
  Spectrum Protect Plus 10.1.17 and later, the default concurrency
  for cloud offload jobs on the vSnap is scaled when there is a
  bandwidth throttle configured for the vSnap. This results in the
  appropriate level of concurrency for throttled cloud offload
  jobs (especially with low throttles), ensuring that the offload
  job does not time out.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels