IT41531: CATALOG BACKUP STOPS WITH "ERROR WRITING DATA FOR COLLECTION `ECDB_MASTER.LOG` TO DISK"

APAR status

• Closed as program error.

Error description

• The IBM Spectrum Protect Plus Appliance scheduled catalog backup
  can stop with the following messages seen in the job log :
  
  SUMMARY,<timestamp>CTGGA2398,Starting job for policy <SLAName>
                               (ID:<SLAId>). id -> <JobId>. IBM
                               Spectrum Protect Plus version
                               10.1.9-2713.
  ...
     INFO,<timestamp>,CTGGA0830,Start protection of Configuration
                                catalog.
  ...
     INFO,<timestamp>,CTGGA2168,Done protection of Configuration
                                catalog. Duration: 2208ms
                                Result:Failed. 1 :
                                2022-06-14T11:00:25.133+0300
                                writing ECDB_master.JOBLOG to
                                archive on stdout
     <UTC timestamp> writing ECDB_master.LOG to archive on stdout
     <UTC timestamp> writing ECDB_master.EVENT to archive on
                     stdout
     <UTC timestamp> writing ECDB_master.ACCESSLOG to archive on
                     stdout
     <UTC timestamp> Failed: archive writer: error writing data
                             for collection 'ECDB_master.LOG' to
                             disk:
                             error reading collection:
                             Executor error during find command ::
                             caused by ::
                             errmsg: "CollectionScan died due to
                                      position in capped
                                      collection being deleted.
                Last seen record id: RecordId(100067024)"
                / Mux ending but selectCases still open 4
     10+2 records in
     10+1 records out
     5253 bytes (5.3 kB) copied  0.473036 s  11.1 kB/s
  ...
    ERROR,<UTC timestamp> IDT,2,CTGGA0825,One or more catalog
                                          tasks failed.
  ...
    ERROR,<UTC timestamp> IDT,2,Encountered an error executing
                                step basicStep1 in job
                                basicOneStepJob
  
  IBM Spectrum Protect Plus Versions Affected:
  IBM Spectrum Protect Plus 10.1.x
  
  Additional Keywords: SPP, SPPLUS, TS009612593, catalog, backup
  

Local fix

• Manually backup the catalog.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.10.2 till 10.1.15       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply the fixing level when available. This problem is       *
  * currently projected to be fixed in IBM Spectrum Protect Plus *
  * level 10.1.15.1 and 10.1.16. Once the fixed version is       *
  * applied, enable the "Lock access log during catalog backup"  *
  * global preference in the IBM Spectrum Protection Plus User   *
  * Interface.                                                   *
  ****************************************************************
  

Problem conclusion

• This is caused by concurrent writes to the audit log while a
  catalog backup is performed, which rotates the audit log data in
  the MongoDB database while the database is being dumped. This
  rotation causes the dump operation to fail, subsequently failing
  the catalog backup. A global preference has been added that can
  lock the audit log during catalog backups, which stops the audit
  log from rotating while the MongoDB database dump is in progress
  and resolves the "CollectionScan died due to position in capped
  collection being deleted" error.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• Catalog
  

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels