IT32541: MAINTENANCE JOB ENDS WITH STATUS "PARTIAL" WITH MESSAGE CTGGA1770 "FAILED TO DESTROY STORAGE SNAPSHOT"

APAR status

• Closed as program error.

Error description

• The Spectrum Protect Plus maintenance job ends with the status
  "Partial" as seen in the following job log example
  :
  SUMMARY,,<timestamp>,,CTGGA2398,Starting job for policy
  Maintenance. id -> <JobID>. IBM Spectrum Protect Plus version
  10.1.5-2181.
  ...
   WARN,,<timestamp>,2,CTGGA1770,Deleting
  storage snapshot for policy <policy_name>
   Failed to destroy
  storage Snapshot message :"Failed to delete snapshot: Command
  failed: cannot create bookmark 'vpool1/fs<xx>#snap<yyyy>':
  bookmark exists"
   type :"SnapshotDeleteError"
  ...
  
  DETAIL,,<timestamp>,,CTGGA0814,Task 2 : Hypervisor Copy
  completed with status PARTIAL
  
  zfs bookmarks are used by
  Spectrum Protect Plus when the snapshot to delete on the source
  vSnap host is in a replication relationship and it is the last
  present.
  This permits to send only the incremental changes of a
  subsequent snapshot on a next replication instead of sending a
  full sized base and snapshot data if no previous snapshot or
  bookmark exists.
  
  In the vSnap log we can see that prior to
  delete the snapshot, a zfs bookmark creation is attempted and
  fails because there is already one existing.
  This aborts the
  snapshot deletion :
  
  [<timestamp>] INFO pid-<12345> vsnap.api
  API request started: DELETE /snapshot/<yyyy>? | Body:
  None
  [<timestamp>] INFO pid-<12345> vsnap.common.util
  Authorized user: vsnap_user
  [<timestamp>] INFO pid-<12345>
  vsnap.zfs Deleting snapshot id <yyyy>
  [<timestamp>] WARNING
  pid-<12345> vsnap.zfs Creating bookmark for snapshot id <yyyy>
  because relationship <replication_ID> may need it for future
  syncs
  [<timestamp>] INFO pid-<12345> vsnap.linux.system
  Executing command: zfs bookmark vpool1/fs105@snap<yyyy>
  vpool1/fs<xx>#snap<yyyy>
  [<timestamp>] ERROR pid-<12345>
  vsnap.linux.system Return code 1: zfs bookmark
  vpool1/fs<xx>@snap<yyyy> vpool1/fs<xx>#snap<yyyy>
  [<timestamp>]
  ERROR pid-<12345> vsnap.linux.system Ouput: ["cannot create
  bookmark 'vpool1/fs<xx>#snap<yyyy>': bookmark exists"]
  
  An
  existing bookmark would for example be possible if a previous
  snapshot deletion process started, created the bookmark but
  then abnormally stopped for any reason before removing the
  snapshot.
  This situation would be seen with the following
  command on the replication source vSnap host :
  
   zfs list -t
  filesystem,snapshot,bookmark | grep fs<xx>
  
  which would list
  for example :
  
   vpool1/fs<xx> 43K 122G 27K
  /vsnap/vpool1/fs<xx>
   vpool1/fs<xx>@snap914 0B - 27K -
  
  vpool1/fs<xx>#snap914 - - - - ==> this is a bookmark using '#'
  while snapshots use'@'
  
  IBM Spectrum Protect Versions
  Affected:
  IBM Spectrum Protect Plus 10.1.4 and 10.1.5
  
  Initial
  Impact: Medium
  
  Additional Keywords: SPP, SPPLUS, TS003530631,
  replication, bookmark, snapshot, delete, expiration
  

Local fix

• 1. Identify with the above maintenance job log message the
  correct bookmark reference.
   In the above example, it is
  'vpool1/fs<xx>#snap<yyyy>'
  2. Manually delete the offending
  bookmark on the replication source vSnap host with the command
  :
   sudo zfs destroy "vpool1/fs<xx>#snap<yyyy>"
  3. Let the next
  maintenance job run proceed with the snapshot deletion.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Spectrum Protect Plus level 10.1.5                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See Error Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in IBM Spectrum Protect Plus level     *
  * 10.1.6. Note that this is subject to change at the           *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• When a snapshot is expired by the Maintenance job and that
  snapshot is the most recent replicated one on the volume, an
  alternate bookmark is created so that future replication
  operations can use the bookmark to perform incremental
  replication of later snapshots. After creating the bookmark if
  the snapshot expiration fails for any other reason, the bookmark
  is left over. The next time SPP Maintenance tries to expire the
  same snapshot, vSnap tries the create the bookmark again but
  this fails because the previous bookmark already exists.
  
  The problem has been resolved by detecting and using existing
  bookmarks, if present, before trying to create new ones.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  SP PLUS

• Fixed component ID

  5737SPLUS

Applicable component levels