IT31886: ORACLE BACKUP TIME OUT ERROR ORA-00245 / RMAN-03009

APAR status

• Closed as program error.

Error description

• The backup of Oracle databases located in an Oracle Real
  Application Clusters (RAC) environment were the Database
  control files are housed in a shared location can
  inttermittently fail with the following messages seen in the
  virgo log :
  
  [<timestamp>] INFO .. <jobID> <DBName>: Completed
  datafile backups; total output bytes = 4639995985920
  (4.22TB)
  [<timestamp>] INFO .. <jobID> <DBName>: Backing up
  archived logs to /mnt/spp/vsnap/vpool1/fs<xx>/<DBName>/arch for
  SCN range 634905906961 to 634916403408.
  [<timestamp>] INFO ..
  <jobID> <DBName>: Completed archived log backups; total output
  bytes = 3060793344 (2.85GB)
  [<timestamp>] INFO .. <jobID>
  <DBName>: Backing up current controlfile.
  [<timestamp>] ERROR
  .. <jobID> [<OracleHostIP>] <DBName>: RMAN-03009: failure of
  backup command on spp1 channel at <timestamp>
  [<timestamp>]
  ERROR .. <jobID> [<OracleHostIP>] <DBName>: ORA-00245: control
  file backup failed; in Oracle RAC, target might not be on
  shared storage
  [<timestamp>] ERROR .. <jobID> Database
  <DBName>: Backup failed: Failed to backup current
  controlfile
  [<timestamp>] INFO .. <jobID> Completed backup
  operation in 269m 54s. 0 database(s) succeeded and 1
  failed.
  
  in the command.log for the control file backup
  :
  
  [<timestamp>] JOBLOG pid:<PID> Thread-<xx> worker_backup:
  <CTGGF0026> <DBName>: Backing up current
  controlfile.
  [<timestamp>] INFO pid:<PID> Thread-<xx>
  writeInputFile: <DBName> set command id to
  'SPP_BACKUP_<SLA_ID>_<jobID>';
  [<timestamp>] INFO pid:<PID>
  Thread-<xx> writeInputFile: <DBName> allocate channel spp1 type
  disk format
  '/mnt/spp/tmp/<DBName>1_spp_backup_<SLA_ID>.ctl';
  [<timestamp>]
  INFO pid:<PID> Thread-<xx> writeInputFile: <DBName> backup as
  copy current controlfile reuse tag
  'SPP_BACKUP_<SLA_ID>';
  [<timestamp>] ERROR pid:<PID>
  Thread-<xx> processOutput: <DBName>: RMAN-03009: failure of
  backup command on spp1 channel at <timestamp>
  [<timestamp>]
  ERROR pid:<PID> Thread-<xx> processOutput: <DBName>: ORA-00245:
  control file backup failed; in Oracle RAC, target might not be
  on shared storage
  [<timestamp>] INFO pid:<PID> Thread-<xx>
  backupControlfile: Traceback (most recent call last):
   File
  "/opt/IBM/SPP/agents_bckadmin/oracleagent/runner.py", line 331,
  in backupControlfile
   Exception: Unexpected messages in command
  output
  [<timestamp>] INFO pid:<PID> Thread-<xx> worker_backup:
  Traceback (most recent call last):
   File
  "/opt/IBM/SPP/agents_bckadmin/oracleagent/runner.py", line 331,
  in backupControlfile
   Exception: Unexpected messages in command
  output
   During handling of the above exception, another
  exception occurred:
   Traceback (most recent call last):
   File
  "/opt/IBM/SPP/agents_bckadmin/oracleagent/backup.py", line 389,
  in worker_backup
   File
  "/opt/IBM/SPP/agents_bckadmin/oracleagent/runner.py", line 338,
  in backupControlfile
   oracleagent.runner.RmanRunnerError:
  Failed to backup current controlfile
  
  both virgo.log and
  command.log files will be found in the job log bundle collected
  from the Spectrum Protect GUI.
  
  IBM Spectrum Protect Versions
  Affected:
  IBM Spectrum Protect Plus 10.1.x
  
  Initial Impact:
  High
  
  Additional Keywords: SPP, SPPLUS, TS003084275
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.5.                      *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in IBM Spectrum Protect Plus level     *
  * 10.1.5 patch1 and 10.1.6. Note that this is subject to       *
  * change at the discretion of IBM.                             *
  ****************************************************************
  

Problem conclusion

• In an Oracle RAC clustered environment, SPP mounts the backup
  volume from vSnap on one of the RAC nodes in order to perform
  the backup. When creating a backup copy of the controlfile, if
  another node is modifying the controlfile at the same time, RMAN
  throws error ORA-00245 indicating that the target of the copy
  operation is not on shared storage.
  
  The issue has been resolved by improving the way SPP performs
  backup of the controlfile in a RAC environment. Instead of
  creating the copy on a non-shared vSnap location, SPP first
  creates a temporary copy in a shared location (the same ASM
  diskgroup where the original controlfile resides) and then
  copies this temporary copy from the shared ASM location to the
  non-shared vSnap location.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels