IT44439: RESTORE OF A VMWARE VIRTUAL MACHINE APPEARS SUCCESSFUL, BUT THE VM IS MISSING DATA OR EXHIBITS OTHER ANOMALOUS BEHAVIOR.

APAR status

• Closed as program error.

Error description

• Under the preceding conditions, no further recovery of the VM is
  possible.The problem occurs during backup. If the data mover
  detects that CBT is not enabled, then the data mover tries to
  enable CBT. If the vCenter user account does not have the
  "toggle disk change tracking" privilege, then the following
  error message is logged to the dsmerror.xxxxxx.log file:
  
  ANS9365E VMware vStorage API error for virtual machine 'XXXX'.
  IBM Spectrum Protect function name : ReconfigVM_USCORETask
  IBM Spectrum Protect file : ..\..\common\vm\
  vmvisdk.cpp (7448)
  API return code : 12
  API error message : SOAP 1.1 fault "":ServerFaultCode
  [no subcode]
  "Permission to perform this operation was denied."
  Detail: <NoPermissionFault xmlns="urn:vim25"
  xsi:type="NoPermission">
  <object type="VirtualMachine">vm-XXXX</object>
  <privilegeId>VirtualMachine.Config.ChangeTracking
  </privilegeId></NoPermissionFault>
  
  At the end of the backup operation, the following message is
  logged to the server activity log, dsmched.xxxxxx.log, or
  console display:
  Total number of virtual machines with no changed data : nnn
  
  If nnn is larger than 0, it is possible that one or more VM
  backups might have been affected by this issue. However, there
  are other reasons that a VM might have no changed data, so this
  message by itself is not conclusive.
  
  IBM Spectrum Protect Versions Affected:
  IBM Spectrum Protect for Virtual Environments Data Protection
  for VMware version 8.1.x on Windows and Linux supported
  platforms
  

Local fix

• Ensure that the vCenter account used for backup has all
  required privileges, as described in this document:
  https://www.ibm.com/support/pages/vcenter-server-privileges-req
  uired-data-protection-vmware-vsphere-gui-and-data-mover
  
  If "Toggle disk change tracking" was missing, then existing
  backups are likely to be unrecoverable. Reset CBT for all
  virtual machines so that the next backup operation is an
  "incremental forever - full" backup. Then continue with
  "incremental forever - incremental" backups.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * IBM Storage Protect for Virtual Environments: Data           *
  * Protection for VMware 8.1.0.0 - 8.1.22.0                     *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR Description                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Immediate action: Check whether the vCenter account that is  *
  * used for backup has the "Toggle disk change tracking"        *
  * privilege.                                                   *
  *                                                              *
  * Important: See https://www.ibm.com/support/pages/node/713157 *
  * for a description of all required vCenter privileges, and    *
  * ensure that the required privileges are enabled.             *
  *                                                              *
  * If "Toggle disk change tracking" is not enabled, then enable *
  * the privilege. After the privilege is enabled, do a one-time *
  * full (-mode=iffull) backup of all VMs. After the full        *
  * backups are completed, resume normal incremental backups.    *
  *                                                              *
  * Note: Incremental backups ("incremental forever -            *
  * incremntal") prior to enabling "Toggle disk change tracking" *
  * are etiher partially or wholly unrecoverable because changes *
  * made since the last full backup ("incremental forever -      *
  * full) are not captured in the incremental backups.           *
  *                                                              *
  * Apply the fixing level when available. This problem is       *
  * projected to be fixed in Data Protection for VMware          *
  * 8.1.23.0. Note that until the fixing level is actually       *
  * available, this information is subject to change at the      *
  * discretion of IBM.                                           *
  ****************************************************************
  

Problem conclusion

• Data Protection for VMware requires that the vCenter account
  used for backup has the "Toggle disk change tracking" privilege.
  This privilege allows the backup data mover to enable Change
  Block Tracking (CBT) after the first full backup of each VM.
  However, the data mover ignored this privilege, and instead ran
  incremental backups of the VM as if CBT was enabled. If CBT was
  not enabled due to the missing privilege, then changed blocks
  were not detected, leading to a loss of backup data. In this
  case, the errors cited in the error description are logged as
  described in the APAR Error Description. Scheduled operations
  show a successful completion status with return code 8.
  
  Beginning in version 8.1.23.0, The data mover is fixed to check
  whether the attempt to enable CBT is successful. If the
  privilege is not held and enabling CBT fails, then incremental
  backups of the VM fail with return code 12. Full (-mode=iffull)
  backups of the VMs are successful and integral, but the ANS9365E
  message that includes the text "Permission to perform this
  operation was denied." is logged and the return code is 8.
  
  The APAR fix makes it more obvious that the "Toggle disk change
  tracking" privilege is missing. It does not prevent the need to
  follow the preceding "Immediate action" steps. If the "Immediate
  action" steps were done before applying the fix, it is not
  necessary to do them again after applying the fix.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• dsmc     dsmcsvc  dsmagent
  

Fix information

• Fixed component name

  TSM FOR VE DP V

• Fixed component ID

  5725TVEVM

Applicable component levels