IT36967: "CTGGA2178 UNEXPECTED ERROR LOADING SOURCE ITEM" ON EXCHANGE DATABASE RESTORE FROM CLOUD OR ARCHIVE

APAR status

• Closed as program error.

Error description

• Starting with version 10.1.3, IBM Spectrum Protect Plus allows
  rerunning under certain conditions failed or partial backup
  jobs.
  When a failed copy to cloud or archive job is rerun, even if it
  completes successfully, the obtained restore point can fail to
  be restored.
  
  In the job log the following messages can be seen :
  
  SUMMARY,<timestamp>,CTGGA2398,Starting job for policy
                                onDemandRestore_<id> (ID:<SLAId>)
                                id -> <JobId>. IBM Spectrum
                                Protect Plus version 10.1.7-3102.
   DETAIL,<timestamp>,CTGGL0055,Recovery policy options:
                { sourceType=Exchange
              selectedSource=[<DBName>]
              sourceSnapshot=[<DBName> - <RestorePointTimeStamp>]
                 restoreType=ondemand
           restoreSourceType=site
               restoreSource=APP_Exchange
               restoreMethod=Test
                 destination=Restore to original instance
     autocleanupOnJobFailure=true
       allowSessionOverwrite=false
  continueWithRestoreOnError=false
                recoveryType=recovery
      continueScriptsOnError=false}
  ...
   DETAIL,<timestamp>,CTGGA1106,Enumerating sources for restore
   DETAIL,<timestamp>,CTGGA1107,Loading source information
   DETAIL,<timestamp>,CTGGA1108,Using URI (https://
                                <SPPHostAddress>/api/application/
                                version/database.<DBId>?from=
                                recovery&embedCopies=true&omitIf
                                NoCopies=true) for policy version
                                (https://<SPPHostAddress>/api/
                                application/version/database.
                                <DBId>?from=recovery&embedCopies=
                                true&omitIfNoCopies=true).
    ERROR,<timestamp>,CTGGA2178,Unexpected error loading source
                                item (<DBName>). Contact support
                                for further assistance.
  
  In the virgo log, the following error stack is observed :
  
  [<timestamp>] ERROR .. Unexpected error loading source item
                         (<DBName>). Contact support for further
                         assistance.
  [<timestamp>] INFO  .. Unhandled exception loading source for
                         recovery job
                         java.lang.NullPointerException: null
     at com.catalogic.ecx.serviceprovider.recovery.common.Catalog
        $StorageVolumeVersion$Links.getRecoveryPoint
        (Catalog.java:5397)
     at com.catalogic.ecx.serviceprovider.recovery.common.Catalog
        $StorageVolumeVersion.getRecoveryPointLink(Catalog.java:
        5363)
     at com.catalogic.ecx.serviceprovider.recovery.application.
        steps.AppLoadCatalogData.loadBackupRawVolume
        (AppLoadCatalogData.java:885)
     at com.catalogic.ecx.serviceprovider.recovery.application.
        steps.AppLoadCatalogData.loadOsVolumeFromStorageVolume
        (AppLoadCatalogData.java:618)
     at com.catalogic.ecx.serviceprovider.recovery.application.
        steps.AppLoadCatalogData.loadCatalogDatabase
        (AppLoadCatalogData.java:362)
     at com.catalogic.ecx.serviceprovider.recovery.application.
        steps.AppLoadCatalogData.loadCatalogDatabase
        (AppLoadCatalogData.java:280)
     at com.catalogic.ecx.serviceprovider.recovery.application.
        steps.AppLoadCatalogData.loadSource(AppLoadCatalogData.
        java:161)
     at com.catalogic.ecx.serviceprovider.recovery.hypervisor.
        LoadCatalogDataStep.loadSources(LoadCatalogDataStep.
        java:410)
     at com.catalogic.ecx.serviceprovider.recovery.hypervisor.
        LoadCatalogDataStep.run(LoadCatalogDataStep.java:143)
     at com.catalogic.ecx.serviceprovider.recovery.hypervisor.
        LoadCatalogDataStep.run(LoadCatalogDataStep.java:129)
     at com.catalogic.ecx.serviceprovider.recovery.application.
        steps.AppLoadCatalogData.run(AppLoadCatalogData.java:126)
     at com.catalogic.ecx.serviceprovider.recovery.application.
        steps.AppLoadCatalogData.run(AppLoadCatalogData.java:80)
     at com.catalogic.ecx.serviceprovider.recovery.common.
        StepChunkExecutor.process(StepChunkExecutor.java:241)
     at com.syncsort.dp.xsb.service.jobmanager.sb.BasicProcessor.
        process(BasicProcessor.java:14)
     at sun.reflect.GeneratedMethodAccessor484.invoke
        (Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke
        (DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:498)
     at org.springframework.aop.support.AopUtils.
        invokeJoinpointUsingReflection(AopUtils.java:333)
     at org.springframework.aop.framework.
        ReflectiveMethodInvocation.invokeJoinpoint
        (ReflectiveMethodInvocation.java:190)
     at org.springframework.aop.framework.
        ReflectiveMethodInvocation.proceed
        (ReflectiveMethodInvocation.java:157)
     at org.springframework.aop.support.
        DelegatingIntroductionInterceptor.doProceed
        (DelegatingIntroductionInterceptor.java:133)
     at org.springframework.aop.support.
        DelegatingIntroductionInterceptor.invoke
        (DelegatingIntroductionInterceptor.java:121)
     at org.springframework.aop.framework.
        ReflectiveMethodInvocation.proceed
        (ReflectiveMethodInvocation.java:179)
     at org.springframework.aop.framework.JdkDynamicAopProxy.
        invoke(JdkDynamicAopProxy.java:213)
     at com.sun.proxy.$Proxy478.process(Unknown Source)
     at org.springframework.batch.core.step.item.
        SimpleChunkProcessor.doProcess(SimpleChunkProcessor.
        java:126)
     at org.springframework.batch.core.step.item.
        SimpleChunkProcessor.transform(SimpleChunkProcessor.
        java:293)
     at org.springframework.batch.core.step.item.
        SimpleChunkProcessor.process(SimpleChunkProcessor.
        java:192)
     at org.springframework.batch.core.step.item.
        ChunkOrientedTasklet.execute(ChunkOrientedTasklet.
        java:75)
     at org.springframework.batch.core.step.tasklet.
        TaskletStep$ChunkTransactionCallback.doInTransaction
        (TaskletStep.java:406)
     at org.springframework.batch.core.step.tasklet.
        TaskletStep$ChunkTransactionCallback.doInTransaction
        (TaskletStep.java:330)
     at org.springframework.transaction.support.
        TransactionTemplate.execute(TransactionTemplate.java:133)
     at org.springframework.batch.core.step.tasklet.
        TaskletStep$2.doInChunkContext(TaskletStep.java:271)
     at org.springframework.batch.core.scope.context.
        StepContextRepeatCallback.doInIteration
        (StepContextRepeatCallback.java:81)
     at org.springframework.batch.repeat.support.RepeatTemplate.
        getNextResult(RepeatTemplate.java:374)
     at org.springframework.batch.repeat.support.RepeatTemplate.
        executeInternal(RepeatTemplate.java:215)
     at org.springframework.batch.repeat.support.RepeatTemplate.
        iterate(RepeatTemplate.java:144)
     at org.springframework.batch.core.step.tasklet.TaskletStep.
        doExecute(TaskletStep.java:257)
     at org.springframework.batch.core.step.AbstractStep.execute
        (AbstractStep.java:200)
     at org.springframework.batch.core.job.SimpleStepHandler.
        handleStep(SimpleStepHandler.java:148)
     at org.springframework.batch.core.job.flow.JobFlowExecutor.
        executeStep(JobFlowExecutor.java:64)
     at org.springframework.batch.core.job.flow.support.state.
        StepState.handle(StepState.java:67)
     at org.springframework.batch.core.job.flow.support.
        SimpleFlow.resume(SimpleFlow.java:169)
     at org.springframework.batch.core.job.flow.support.
        SimpleFlow.start(SimpleFlow.java:144)
     at org.springframework.batch.core.job.flow.FlowJob.
        doExecute(FlowJob.java:134)
     at org.springframework.batch.core.job.AbstractJob.execute
        (AbstractJob.java:306)
     at org.springframework.batch.core.launch.support.
        SimpleJobLauncher$1.run(SimpleJobLauncher.java:135)
     at java.util.concurrent.ThreadPoolExecutor.runWorker
        (ThreadPoolExecutor.java:1149)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run
        (ThreadPoolExecutor.java:624)
     at java.lang.Thread.run(Thread.java:748)
  
  IBM Spectrum Protect Plus Versions Affected:
  IBM Spectrum Protect Plus 10.1.3 and later
  
  Initial Impact: Medium
  
  Additional Keywords: SPP, SPPLUS, TS005291813, offload, backup,
                       re-run
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.5, 10.1.6, 10.1.7 and   *
  * 10.1.8                                                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * see ERROR Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply the fixing level when available. This problem is       *
  * currently projected to be fixed in IBM Spectrum Protect Plus *
  * level 10.1.8 patch1 and 10.1.9.                              *
  * Note that this is subject to change at the discretion of     *
  * IBM.                                                         *
  ****************************************************************
  

Problem conclusion

• Fixed a backup cataloging issue when one item in the job was
  completed while another item failed initially but completed with
  a rerun. The issue was causing intermittent failures of restore
  of an item that was protected by the rerun option. With the fix
  the restore of the item that was protected by rerun will
  complete successfully.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels