IT35094: CTGGR0026 'AN UNEXPECTED ERROR OCCURRED WHEN PROCESSING THE JOB' ON MICROSOFT OFFICE 365 MAILBOX RESTORE

APAR status

• Closed as program error.

Error description

• IBM Spectrum Protect Plus protection support for Microsoft
  Office 365 started with version 10.1.5.
  The Microsoft Office 365 Mailbox restore attempted with IBM
  Spectrum Protect Plus might stop with the following messages
  seen in the job log :
  
  ERROR,,<timestamp>,CTGGR0009,[<SPP-O365-ProxyHost>]
         EXCHANGE MAILBOX RESTORE - <User_email> - Task ended
         with status: failed.
  ERROR,,<timestamp>,CTGGR0026,[<SPP-O365-ProxyHost>]
         EXCHANGE MAILBOX RESTORE - <User_email> - An unexpected
         error occurred when processing the job.
  
  In the CloudAgent log, all threads to the Exchange Web Services
  (EWS) are timing out.
  Example for one thread :
  
  <timestamp> [INF] [taskId=<TaskID>] [threadId=xx]
         SendRequestWithRestResponseAsync Request Completed: OK
  ...
  <timestamp> [ERR] [taskId=<TaskID>] [threadId=xx]
         ExchangeMailboxRestoreFeature --- Sender has thrown an
         exception
  System.AggregateException: One or more errors occurred.
        (The request failed. The operation has timed out.)
   ---> Microsoft.Exchange.WebServices.Data.
        ServiceRequestException: The request failed.
        The operation has timed out.
   ---> System.Net.WebException: The operation has timed out.
     at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult
        asyncResult)
     at System.Net.WebRequest.<>c.<GetResponseAsync>b__68_2
        (IAsyncResult iar)
     at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic
        (IAsyncResult iar, Func`2 endFunction, Action`1
        endAction, Task`1 promise, Boolean
        requiresSynchronization)
     --- End of stack trace from previous location where
         exception was thrown ---
     at Microsoft.Exchange.WebServices.Data.EwsHttpWebRequest.
        Microsoft.Exchange.WebServices.Data.IEwsHttpWebRequest.
        GetResponse()
     at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.
        GetEwsHttpWebResponse(IEwsHttpWebRequest request)
     --- End of inner exception stack trace ---
     at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.
        GetEwsHttpWebResponse(IEwsHttpWebRequest request)
     at Microsoft.Exchange.WebServices.Data.ServiceRequestBase.
        ValidateAndEmitRequest()
     at Microsoft.Exchange.WebServices.Data.
        SimpleServiceRequestBase.InternalExecuteAsync()
     at Microsoft.Exchange.WebServices.Data.
        MultiResponseServiceRequest`1.ExecuteAsync()
     at Microsoft.Exchange.WebServices.Data.Item.InternalCreate
        (FolderId parentFolderId, Nullable`1 messageDisposition,
        Nullable`1 sendInvitationsMode)
     --- End of inner exception stack trace ---
     at System.Threading.Tasks.Task.Wait(Int32
        millisecondsTimeout, CancellationToken cancellationToken)
     at System.Threading.Tasks.Task.Wait()
     at CloudAgent.Core.Office365.Providers.EWSProvider.
        RestorEmail(FolderId parentFolderId, Byte[] mimeContent)
        in C:\BuildAgent\work\cloud4Cloud\Release\src\CloudAgent.
        Core\Office365\Providers\EWSProvider.cs:line 120
     at CloudAgent.Core.Office365.Features.
        ExchangeMailboxRestoreFeature.<>c__DisplayClass17_0.
        <Sender>b__0(ValueTuple`4 current) in C:\BuildAgent\work\
        cloud4Cloud\Release\src\CloudAgent.Core\Office365\
        Features\ExchangeMailboxRestoreFeature.cs:line 303
  ...
  <timestamp> [INF] [taskId=<TaskID>] [threadId=yy] Task <TaskID>
         finished. Status: "FAILED", Message: Processed 0 of
         <zzzz> EMAIL and <aa> of <aa> DIRECTORY,
         ErrorCodes: 9999: 5
  
  This can happen when the connection to the Microsoft Exchange
  Web Services is timing out.
  
  IBM Spectrum Protect Plus Versions Affected:
  IBM Spectrum Protect Plus 10.1.5 and 10.1.6
  
  Initial Impact: Medium
  
  Additional Keywords: SPP, SPPLUS, TS004519209, timeout
  

Local fix

• Retry the failing operation.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  *  IBM Spectrum Protect Plus level 10.1.6, 10.1.7 and 10.1.8   *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed IBM Spectrum Protect Plus level        *
  * 10.1.9. Note that this is subject to change at the           *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• In some cases, connections timeout could be a fault of the
  incorrect connection management, the issue has been fixed.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels