IT36784: RESTORE OF MICROSOFT OFFICE 365 OBJECTS ENDING WITH 'UNEXPECTED ERROR' BECAUSE OF 'SERVERBUSYEXCEPTION'

APAR status

• Closed as program error.

Error description

• Restore of Microsoft Office 365 objects ending with
  'unexpected error' because of 'ServerBusyException'
  
  Error Description:
  
  Starting with version 10.1.5, IBM Spectrum Protect Plus can
  protect Microsoft Office 365 objects.
  A restore can stop with the following messages seen in the job
  log :
  
  SUMMARY,Thu <timestamp>,CTGGA2398,Starting job for policy
                                    onDemandRestore_<Id>
                                    (ID:<xxxx>). id -> <JobId>.
                                    IBM Spectrum Protect Plus
                                    version 10.1.7-3051.
   DETAIL,Thu <timestamp>,CTGGL0055,Recovery policy options:
    {      sourceType=Office 365
       selectedSource=[Mailbox]
       sourceSnapshot=[Mailbox - Outlook/<MailBoxUserEmail> -
                       <RestorePointTimeStamp>]
          restoreType=ondemand
    restoreSourceType=site
        restoreSource=<SiteName>
          restorePath=<PathName>
        restoreMethod=Restore to other account
             tenantID=<TenantId>
     usernameSelected=<TargetUserEmail>}
  ...
   DETAIL,<timestamp>,CTGGA4000,[<ProxyHostAddress>] Unix Host
                                Agent <SPPO365AgentVersion>
                                running as <ProxyHostUser> for
                                appdiscovery (task ID: <TaskId>)
   DETAIL,.. Hostname: LPRD-PROXYSPP01.caser.local / Operating
             System: Red Hat Enterprise Linux 8.1 x86_64 /
             Python: 3.6.9
  ...
     INFO,.. EXCHANGE MAILBOX RESTORE - <MailBoxUserEmail> -
             - 1111 items processed
     INFO,.. - 2222 items processed
     INFO,.. - xxxx items processed
     INFO,.. - xxxx items processed
    ERROR,.. - Task ended with a failed status.
    ERROR,.. - An unexpected error occurred when processing the
               job.
  SUMMARY,.. - processed items: xxxx (success: yyyy  skipped: 0
               failed: zzz)
  ...
  SUMMARY,<timestamp>,CTGGA2340,Job policy onDemandRestore_<Id>
                                completed in xx min(s) yy sec(s)
                                and ended with status FAILED.
                                id <JobId>
  
  in the CloudAgent.log found in the job log bundle, the
  following error will be seen for one or more objects :
  
  <timestamp> [ERR] [taskId=<TaskId>] [threadId=xx]
                    ExchangeMailboxRestoreFeature -> Sender has
                    thrown an exception
  Microsoft.Exchange.WebServices.Data.ServerBusyException:
     The server cannot service this request right now.
     Try again later.
     at ...
  <timestamp> [INF] [taskId=<TaskId>] [threadId=xx] Item
                    '<ItemId>', action '"Restore"', status
                    'failed, moved to queue'
  
  Several retries will be seen :
  <timestamp> [ERR] [taskId=<TaskId>] [threadId=xx]
                    ExchangeMailboxRestoreFeature -> Sender has
                    thrown an exception
  Microsoft.Exchange.WebServices.Data.ServerBusyException:
     The server cannot service this request right now.
     Try again later.
  ...
  <timestamp> [ERR] [taskId=<TaskId>] [threadId=xx] Item
                    '<ItemId>', action '"ReRestore"', status
                    'failed (#5)'
  
  the task '<TaskId>' ends with :
  <timestamp> [INF] [taskId=<TaskId>] [threadId=xx]
                    Task progress: 100%
  <timestamp> [INF] [taskId=<TaskId>] [threadId=xx]
                    Task <TaskId> finished. Status: "FAILED",
                    Message: Processed yyyy of xxxx EMAIL and zz
                    of zz DIRECTORY, ErrorCodes: 9999: 940
  
  IBM Spectrum Protect Plus Versions Affected:
  IBM Spectrum Protect Plus 10.1.5 and higher
  
  Initial Impact: Medium
  
  Additional Keywords: SPP, SPPLUS, TS005545522, mailbox restore,
                       busy, unknown
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.7 and 10.1.8            *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in IBM Spectrum Protect Plus level     *
  * 10.1.9. Note that this is subject to change at the           *
  * discretion of IBM                                            *
  ****************************************************************
  

Problem conclusion

• This problem has been fixed in a way that the M365 agent will
  retry to download objects after the wait time specified by
  Exchange Online.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• O365
  

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels