APAR status
Closed as program error.
Error description
vCenter 6.5 doesn't return an error after Data Protection for VMware tries to get CBT data when CBT was enabled with a pre-existing snapshot. In case CBT was enabled with pre-existing snapshots, the CBT data returned by VMware must be considered invalid: - it may be empty causing no data being sent to the server. - it can represent some random blocks. In this case, invalid data is sent to the server. In both cases, the backup can not be considered as a valid backup. CBT can be enabled manually by Data Protection for VMware, by other products or manually by the user. However, it is impossible to detect afterwards if a snapshot existed when CBT was enabled or re-enabled. IBM Spectrum Protect Versions Affected: IBM Spectrum Protect for Virtual Environments - Data Protection for VMware 8.1.x on Windows and Linux Customer/L2 Diagnostics: In case empty CBT data is returned, the backup summary will show no disks being backed up and only configuration information is sent to the server. Example: Protect> backup vm 'testvm' Full on-demand BACKUP VM of virtual machines 'testvm'. Backup VM command started. Total number of virtual machines to process: 1 Accessing as node: DC_node Starting Full VM backup of VMware Virtual Machine 'testvm' mode: 'Incremental Forever - Incremental' target node name: 'DC_node' data mover node name: 'DM_node' application protection type: 'VMware' application(s) protected: 'n/a' Creating "VMware Tools" snapshot for virtual machine 'testvm' Backing up Full VM configuration information for 'testvm' 31,451 VM Configuration [Sent] >>>> Note missing disks during the backup of the VM <<<<< Successful Full VM backup of VMware Virtual Machine 'testvm' mode: 'Incremental Forever - Incremental' target node name: 'DC_node' data mover node name: 'DM_node' Statistics for Virtual Machine 'testvm'. Total number of objects inspected: 1 Total number of objects backed up: 1 ... Total number of bytes inspected: 40.00 GB Total number of bytes processed: 31.25 KB ... Removing snapshot for virtual machine 'testvm' Backup VM command complete Total number of virtual machines backed up successfully: 1 virtual machine testvm backed up to nodename DC_node Total number of application protection warnings: 0 Total number of virtual machines failed: 0 Total number of virtual machines skipped: 0 Total number of virtual machines processed: 1 When the vCenter returns invalid CBT data, disk data will be sent to the server. This will look like a normal, successful backup. Review a trace with service, vm traceflags: Only in case Data Protection for VMware tries to enable CBT and detect an existing snapshot while enabling CBT, the following line is added to the trace: "vmVddkFullVMPrePareToOpenVMDKs: pre-existing snapshot are not recommend when enabling CBT". *NOTE: If CBT was enabled outside Data Protection for VMware, it is impossible to detect if a snapshot existed or not. Backup data is invalid in this case, but this can not be detected by Data Protection for VMware. Products affected: IBM Spectrum Protect for Virtual Environments: Data Protection for VMware version 8.1 on Windows x64 and Linux x86_64 platforms. This problem also affects IBM Spectrum Protect Backup-Archive Client which is the data mover for Data Protection for VMware version 7.1. If you are using Backup-Archive Client version 7.1 refer to APAR IT28068 Initial Impact: high Additional Keywords: TSM, Tivoli, Storage, Manager, TDP, TDP4VE, TS000844751
Local fix
Verify if no snapshot exists when enabling CBT for a Virtual Machine.
Problem summary
**************************************************************** * USERS AFFECTED: * * IBM Spectrum Protect for Virtual Environments Data * * Protection for VMware versions 7.1 and 8.1 running on all * * Microsoft Windows x64 and Linux x86 platforms. * **************************************************************** * PROBLEM DESCRIPTION: * * See ERROR DESCRIPTION * **************************************************************** * RECOMMENDATION: * * This issue is projected to be fixed in the Data Protection * * for VMware Data Mover version 8.1.7 on Windows x64 and Linux * * x86 platforms. * * Note 1: This is subject to change at the discretion of IBM. * ****************************************************************
Problem conclusion
The issue is due to a bug in vSphere 6.5 and higher: when CBT is enabled with a pre-existing snpashot the QueryChangedDiskAreas might return invalid data. https://kb.vmware.com/kb/1031873. The code has been updated to perform extra verification of CBT data. Also, a new Vmnocbtcontinue option has been added to specify whether to back up a virtual machine (VM) without using the change block tracking function when one or more snapshots already exist on the VM and change block tracking must be enabled or reset.
Temporary fix
Comments
The issue is due to a bug in vSphere 6.5 and higher: when CBT is enabled with a pre-existing snpashot the QueryChangedDiskAreas might return invalid data. https://kb.vmware.com/kb/1031873
APAR Information
APAR number
IT26261
Reported component name
TSM FOR VE DP V
Reported component ID
5725TVEVM
Reported release
81W
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2018-09-14
Closed date
2019-02-19
Last modified date
2019-05-21
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Modules/Macros
dsmagent
Fix information
Fixed component name
TSM FOR VE DP V
Fixed component ID
5725TVEVM
Applicable component levels
[{"Line of Business":{"code":"LOB26","label":"Storage"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SS8TDQ","label":"Tivoli Storage Manager for Virtual Environments"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"81W"}]
Document Information
Modified date:
14 February 2021