IBM Tivoli Storage Manager Disaster Recovery Testing

Answer

The two main reasons are because backups didn't happen and backups can't be used. A few miscellaneous issues crop up as well.

Different technologies (backups, disk mirroring, virtualization) reduce one or another of these causes, but there's no magic technological bullet that eliminates all of them.

• Backups didn't happen.
  • They weren't made at all.
    That's why people have backup-monitoring software like Servergraph, Bocada. However, we know one story where operators "learned" that the missed backup alerts were "spurious" and should be ignored. This misconception didn't come to light until recovery-failure time.
  • Node was never registered.
    If Tivoli Storage Manager doesn't even know a node exists, you can't recover it from Tivoli Storage Manager. This problem is becoming more prevalent now that VMware machines are popping up everywhere.
  • DOMAIN statement ignores drive E:
    User doesn't want to bother backing up C: because it's a standard corporate image. So they change DOMAIN from ALL_LOCAL to D: Then, 6 months later, they add an E: drive. Tivoli Storage Manager quietly ignores E: forever.
  • Includes/ excludes skipped an important file.
    The inclexcl file uses patterns (regexes). There are patterns for both include and exclude that apply throughout and the results is difficult to understand. It's fairly common for users (or even experienced Tivoli Storage Manager administrators) to have an exclude list that doesn't do what you think it does. 
  • Database backup breaks log sequence.
    If someone (the DBA) does a full backup in mid-week, the database backups that use full-weekend + incremental-weekdays (Grandfather-Father-Son) scheme aren't restorable. (They are restorable from the full backup. However, if you are the storage administrator, the DBA might not tell you that they completed the restore.)

• Backups cannot be used
  • Tape volumes are marked "unavailable"
    Tivoli Storage Manager usually marks a tape volume as bad when the tape drive has a problem. For this and other reasons, some of your backup tapes might not be usable unless you set them back to read/write. Unfortunately, bad tape volumes also behave this way so it's hard to tell the difference between bad-drive and bad-tape.
  • Copy pool volumes get damaged during off-site Disaster Recovery tests
    Off-site Disaster Recovery tests that make you move your copy pool tapes to Sterling Forest put them at risk of being dropped, overheated, or otherwise damaged in a hostile environment. So your Disaster Recovery test can succeed, at the same time it causes a future problem.
  • Disaster Recovery Plan is missing
    Without a list of the required tape volumes and device configurations, you cannot restore the Tivoli Storage Manager server. The "prepare" command generates a good Disaster Recovery plan. However, the Disaster Recovery Manager (DRM) might not be licensed, or the "prepare" command might not be scheduled, or the drplan file might not be moved off-site.
  • Tivoli Storage Manager database backup is missing
    In one case, someone pruned the volhist.txt file, so Tivoli Storage Manager could not figure out what volume held the latest Tivoli Storage Manager database backup.
  • User lost their encryption key
    If the dsmc client is set up to use encryption, the user is responsible for knowing the encryption key. If they lose it, they can't restore any encrypted files.

• Miscellaneous Issues
  • Inadequate performance
    Flood-and-fire disasters leave the recovered business with a large catch-up load. Performance of the Disaster Recovery site needs to be significantly higher than the normal production site, at least until the catch-up period is over.
  • Rolling disaster
    Though remote mirrors can help you to achieve high RPOs / RTOs, they are not a substitute for backups. The database administrator whose maintenance script runs amok might need to restore from last week's backup, because the mirrored volume can corrupt simultaneously.
  • Interdependent systems
    The Tier 1 system that you can restore might depend on a system you consider Tier 3. You might have valid backups for the first, but not for the second.