IT37272: "RETRIESEXCEEDEDERROR: MAX RETRIES EXCEEDED" ON RESTORE FROM ARCHIVE FOR VMS WITH DISKS OF 1TB OR MORE

APAR status

• Closed as program error.

Error description

• When using the following IBM Spectrum Protect Plus vSnap
  restore from archive option value 'TRUE' with the command :
  vsnap system pref set --name archiveDownloadBeforeExtraction
  --value true
  as workaround referenced for APAR IT34012, the restore from
  archive can stop after many hours for guests disks of 1TB or
  more.
  
  In the job log the error can be seen after many hours of
  processing :
  
  SUMMARY,<timestamp>,CTGGA2398,Starting job for policy
                                onDemandRestore_12345678912
                                (ID:1234). id -> <JobId>. IBM
                                Spectrum Protect Plus version
                                10.1.7-3102.
  ...
     INFO,<timestamp>,2,CTGGA2589,Creating clones of vSnap
                                  volumes.
   DETAIL,<timestamp>,2,CTGGA2173,Cloning volume
                                  (<vSnapVolumeName>) from
                                  snapshot (<SnapshotName>) of
                                  archive volume
                                  (<ArchivedVolumeName>)
   DETAIL,<timestamp>,2,CTGGA0980,Starting to create volume clone
                                  (<vSnapVolumeName>) from
                                  snapshot (<SnapshotId>) using
                                  session ID
                                  (<ReplicationSessionId>).
     INFO,<timestamp>,2,CTGGA2287,Archive volume clone creation
                                  in progress: Clone
                                  (<vSnapVolumeName>) Data
                                  transferred (37893345871)
                                  Status message (Data transfer
                                  in progress. Downloaded
                                  40106.95 MB. Throughput:
                                  68.96 MB/s)
  ...
     INFO,<timestamp>,2,CTGGA2287,Archive volume clone creation
                                  in progress: Clone
                                  (<vSnapVolumeName>) Data
                                  transferred (4312463141953)
                                  Status message (Data transfer
                                  in progress. Downloaded
                                  4148823.25 MB. Throughput:
                                  18.09 MB/s)
    ERROR,<timestamp>,2,CTGGA0986,Async volume clone creation
                                  failed for volume
                                  (<ArchivedVolumeName>) snapshot
                                  (<SnapshotName>) error
                                  (RetriesExceededError: Max
                                  Retries Exceeded).
    ERROR,<timestamp>,2,CTGGA1110,No selected item is recoverable
  
  In the replication log, the actual error is displayed and after
  5 retries, the replication session is aborted :
  
  [<timestamp>] INFO pid-1234 vsnap.repld  Session
                                           <ReplicationSessionId>
                                           :worker started
  ...
  [<timestamp>] INFO pid-1234 vsnap.archive.mover
                     Waiting for files from the archive provider.
                     This may take several hours.
  .. Started restoring 16 files from from 16 objects
  .. Files will be downloaded before extracting.
  .. Session <ReplicationSessionId>: size received = 37893345871
                                     (35.29GB)
  .. Session <ReplicationSessionId>: message = Data transfer in
             progress. Downloaded 40106.95 MB. Throughput:
             159.60 MB/s
  ... <after a long time>
  [<timestamp>] WARNING pid-1234 vsnap.linux.system
        Ouput: ['star: Trying to access sparse aray beyond end
                       (index <xxxx>).',
                "star: Error writing'<VMGuestDiskName>-flat.vmdk
                       '.",
                'star: Tar file too small (amount: 0 bytes).',
                'star: Unexpected EOF on input.',
                'star: Cannot recover from error - exiting.']
  [<timestamp>] WARNING pid-1234 vsnap.archive.util
        Download file /vsnap/vpool1/fs123/a1b2c3/<VMName>.vm-
        <VMManagedObjectId>/<VMGuestDiskName>-flat.vmdk failed,
        retrying 1/5.
        Error: Command failed: star: Trying to access sparse
                                     array beyond end
                                     (index <xxxx>).;
              star: Error writing '<VMGuestDiskName>-flat.vmdk'.;
              star: Tar file too small (amount: 0 bytes).;
              star: Unexpected EOF on input.;
              star: Cannot recover from error - exiting.
  ...
  [<timestamp>] INFO pid-1234 vsnap.common.model  Session
                     <ReplicationSessionId>: message = Data
                     transfer in progress. Downloaded 3025245.12
                     MB. Throughput: 20.05 MB/s
  [<timestamp>] WARNING pid-1234 vsnap.linux.system  Return code
                255:
        star x -C '/vsnap/vpool1/fs123/a1b2c3/<VMName>.vm-
        <VMManagedObjectId>/tmp_<zzzzzzzzzz>'
        --compress-program="lz4" -sparse -silent -no-statistics
        -f '/vsnap/vpool1/fs123/a1b2c3/<VMName>.vm-
        <VMManagedObjectId>/tmp_<zzzzzzzzzz>/<VMGuestDiskName>-
        flat.vmdk.tar.lz4'
  [<timestamp>] WARNING pid-1234 vsnap.linux.system
        Ouput:
        ['star: Trying to access sparse aray beyond end
                (index <xxxx>).',
         "star: Error writing '<VMGuestDiskName>-flat.vmdk'.",
         'star: Tar file too small (amount: 0 bytes).',
         'star: Unexpected EOF on input.', 'star: Cannot recover
                from error - exiting.']
  [<timestamp>] WARNING pid-1234 vsnap.archive.util
        Download file /vsnap/vpool1/fs123/a1b2c3/<VMName>.vm-
        <VMManagedObjectId>/<VMGuestDiskName>-flat.vmdk failed
        after 5 attempts.
        Error: Command failed:
        star: Trying to access sparse aray beyond end
              (index <xxxx>).;
        star: Error writing '<VMGuestDiskName>-flat.vmdk'.;
        star: Tar file too small (amount: 0 bytes).;
        star: Unexpected EOF on input.;
        star: Cannot recover from error - exiting.
  
  The archived data is consistent but the vSnap fails to get it
  restored as expected.
  
  | MDVRPARTL 5737SPLUS 10.1.6 | IT34012
  
  IBM Spectrum Protect Plus Versions Affected:
  IBM Spectrum Protect Plus 10.1.7 and later
  
  Additional Keywords: SPP, SPPLUS, TS005816967, restore, start,
                       compression, IT34012
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.7 and 10.1.8            *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in IBM Spectrum Protect Plus level     *
  * 10.1.9. Note that this is subject to change at the           *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• The problem occurred because files larger than 1TB which were
  uploaded to archive as multiple parts were not reconstructed
  correctly during restore. The issue has been resolved by
  implementing code fixes to ensure parts of large files are
  correctly reconstructed and uncompressed during restore from
  archive to vSnap.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• vSnap    Archive
  

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels