IT36641: CTGGA2325 "WAITING FOR REMOTE OPERATION TO COMPLETE" SEEN EVERY 30 MINUTES AND BACKUP JOB DOES NOT COMPLETE

APAR status

• Closed as program error.

Error description

• Using IBM Spectrum Protect Plus, the backup of a guest with
  File Indexing enabled can successfully send the backup data but
  appears to wait forever for the File Indexing process
  completion.
  
  In the backup job log the following will be seen :
  
  SUMMARY,<timestamp>CTGGA2399,Starting job for policy BACKUP
                               with job name <SLAName>
                               (ID:<SLAId>). id -> <JobId>. IBM
                               Spectrum Protect Plus version
                               10.1.6-2040.
  ...
     INFO,<timestamp>,CTGGA0494,File Discovery in progress for
                                VM <GuestName>
   DETAIL,<timestamp>,CTGGA0576,Initializing backup of vm:
                                <GuestName> (host: <ESXiAddress>)
                                data center: <DataCenterName>
  ...
   DETAIL,<timestamp>,CTGGA0671,Backing up VM (<GuestName>) from
                                remote proxy (IP: <vADPAddress>
                                Host name: <vADPHostName>)
     INFO,<timestamp>,CTGGA0252,Starting file catalog discovery
                                on the hypervisor <GuestName>.
  ...
   DETAIL,<timestamp>,CTGGA0074,
           Protected vm: <GuestName>
           proxy: <vADPAddress>
           host: <ESXiAddress>
           vsnap: <vSnapAddress>/<vSnapVolumeName>
           transport:[nbdssl] [volume:<DatastoreName>]
   DETAIL,<timestamp>,2,CTGGA0100,Collecting details of vms and
                                  volumes
   DETAIL,<timestamp>,2,CTGGA2325,Waiting for remote operation to
                                  complete on server <GuestIP>
  
  The last message is repeated indefinitely every 30 minutes.
  
  In the virgo log an exception message will be seen only as
  informational :
  
  [<timestamp>] INFO .. File download failed on host(<GuestIP>).
                        Error({}) com.jcraft.jsch.SftpException:
                        No such file
                        at ...
  [<timestamp>] INFO .. Failed to process status/messages from
                        new log file. e=2: No such file
  ...
  [<timestamp>] INFO .. Waiting for remote operation to complete
                        on server <GuestIP>
  
  IBM Spectrum Protect Plus Versions Affected:
  All supported versions of IBM Spectrum Protect Plus 10.1.x
  
  Initial Impact: Medium
  
  Additional Keywords: SPP, SPPLUS, TS004710125, fileindexing,
                       catalog, file restore
  

Local fix

• Ensure the WinRM service for Windows guest and the ssh/sftp
  service for Linux guests are in a healthy state.
  Eventually restart these services or restart the guest to reset
  connectivity with the IBM Spectrum Protect Plus host so File
  Indexing can complete successfully.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.x.                      *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply the fixing level when available. This problem is       *
  * projected to be fixed in IBM Spectrum Protect Plus level     *
  * 10.1.8 patch1 and 10.1.9. Note that this is subject to       *
  * change at the discretion of IBM.                             *
  ****************************************************************
  

Problem conclusion

• A change was made in the code to wait for remote execution to
  finish on the guest VM. This change prevents certain cases where
  an exception is generated but the task is never marked as
  finished.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels