APAR status
Closed as program error.
Error description
During an IBM Spectrum Protect Plus offload operation, it can fail with the following error messages: WARN,...,CTGGA1915,Error occured on Storage Vsnap replication Please check Vsnap Storage logs. ERROR,...,CTGGA0309,Copy failed for snapshot (ID: nnn) from source [server: <hostname> volume: spp_<volume name> snapshot: spp_<snapshot name>] to target [server: <hostname> volume: <volume name>]. Error: Exception: Failed to create gateway device: Checksum failed for /opt/vsnap-data/offload_<id>/bin_blockmap. Expected <ID> was <ID> ERROR,...,Skipping remaining snapshots for volume <hostname> :spp_<volume name> due to unrecoverable error for Vsnap session <session ID> This occurs when the checksum of the blockmap metadata cached in the offload cache area (/opt/vsnap-data/) does not the match expected value recorded in the vSnap configuration DB. This checksum mismatch typically occurs due to errors in updating the DB that occurred during a previous run of the offload job. IBM Spectrum Protect Plus Versions Affected: 10.1.4 Customer/L2 Diagnostics (If Applicable) N/A Initial Impact: Low Additional Keywords: TS002596475 BLOCKMAP
Local fix
As a workaround, delete the offload directory using the command: rm -rf /opt/vsnap-data/offload_<id> Once deleted, try another incremental offload, which will download the proper blockmap to the system from the cloud.
Problem summary
**************************************************************** * USERS AFFECTED: * * IBM Spectrum Protect Plus level 10.1.4. * **************************************************************** * PROBLEM DESCRIPTION: * * See Error Description. * **************************************************************** * RECOMMENDATION: * * Apply fixing level when available. This problem is currently * * projected to be fixed in IBM Spectrum Protect Plus level * * 10.1.5. Note that this is subject to change at the * * discretion of IBM. * ****************************************************************
Problem conclusion
The problem occurred due to problems with database locking. At the end of a successful cloud offload vSnap attempts to update some configuration database tables record the status of the offload session. This includes storing the checksum of blockmap metadata files in the configuration DB. If the database is under load during this step, the offload process may fail to save the new checksum. As a result the DB still contains an older checksum which is no longer valid. On subsequent offload attempts, vSnap attempts to verify the checksum of the metadata against the value stored in the DB. The offload fails because the value stored in the DB is out of date. The problem has been resolved by improving the database locking process as well as improving the wait-and-retry logic to ensure that the configuration DB is correctly updated at the end of a successful offload.
Temporary fix
Comments
APAR Information
APAR number
IT30343
Reported component name
SP PLUS
Reported component ID
5737SPLUS
Reported release
A14
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2019-09-20
Closed date
2019-10-21
Last modified date
2019-10-21
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
SP PLUS
Fixed component ID
5737SPLUS
Applicable component levels
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSNQFQ","label":"IBM Spectrum Protect Plus"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"A14","Line of Business":{"code":"LOB26","label":"Storage"}}]
Document Information
Modified date:
30 January 2024