Backup and restore

IBM® Db2® Event Store allows you to backup and restore your database to and from a Cloud-based Object Storage, or filesystem which use NFS or hostPath for shared storage. This can be used to protect your database from disaster recovery scenarios, or to restore the database to a specific point in time.

Note: To backup IBM Watson™ Studio Local specific elements, refer to Back up and restore Watson Studio Local.
This utility needs the database to be offline before completing a backup, which means that the database will not be able to serve any requests while the backup completion step is in progress. The database will be brought back online as soon as the backup has finished. The backup operation includes three steps that can be invoked independently:
  • Initialization

    The validation of the backup name is performed. This operation is fully done online.

  • Update
    The backup of shared storage is performed. Due to the immutable state of the data in shared storage, this operation can be done online. This step can be repeated multiple times, and each time will incrementally update the backup.
    Note: The backup is not complete and usable until the backup completion is finished.
  • Completion
    This is the final step of the backup and performs a final update before proceeding to turning the database offline to complete the offline operations that are required for a consistent backup. Once the backup is completed, the database is brought back online, which can now serve client requests.
    Note: Once a backup is brought to this last step, it can not be modified. A new backup must be started to make changes to it.

The restore operation is fully done offline. The restore process will validate the backup and then proceed to turn the database offline to initiate the restoring of data. Once the restore operation is complete, the database is brought back online and the your access is restored.

Encountered Scenarios

  • Filesystem to filesystem backup and restore

    For backing up a filesystem to a filesystem, the backup will be stored in the external shared directory that is mounted across all nodes under the $INSTALL_PATH/eventstore/engine/utils/external_db. For more information on a shared-file system, refer to Connecting to an external shared-file system for utilities.

    The backup will create a directory called backup in that path, that can be overridden using the --backup-path option. Within this new directory, each backup will be stored in its own sub-directory using the backup name, and contains the packages corresponding to each of the components of the backup.

  • COS to COS backup and restore

    For backing up to a COS, the backup will be stored using the path: backup/<backup name>/. This new file path will contain a copy of the shared data together with the directory structure from the source directory maintaining the same naming convention, and will also contain a copy of the all the packages created for each of the components of an IBM Db2 Event Store database.

Environment to run backup and restore

The backup and restore operations run within the containers that are part of the Db2 Event Store deployment. To start both of these, you must use the tools container as an entry point. In order to use this utility without going into the tools container, you can run the following on any of the your cluster's host machine depending on the your deployment:
Note: For more information, refer to IBM Db2 Event Store utilities setup and usage as the setup and installation is similar.
For Standalone/IBM Watson® Studio deployments, run the following:
  • For backup
    TOOLS_CONTAINER=$(kubectl get pods -n dsx | grep eventstore-tenant-tools | awk {'print $1'}) && ORIGIN_IP=`hostname -i | awk '{print $1}'` 
backup(){ kubectl exec $TOOLS_CONTAINER -n dsx -- /eventstore/tools/backup $@; }
  • For restore
    TOOLS_CONTAINER=$(kubectl get pods -n dsx | grep eventstore-tenant-tools | awk {'print $1'}) && ORIGIN_IP=`hostname -i | awk '{print $1}'` 
restore(){ kubectl exec $TOOLS_CONTAINER -n dsx -- /eventstore/tools/restore $@; }
For IBM Cloud Pak for Data deployments, refer to Identifying the deployment ID of Db2 Event Store in a IBM Cloud Pak for Data deployment first, before running the following commands. Also note that you must be logged on to RedHat OpenShift and select the current project of the desired deployment through oc project. For example: 
DEPLOYMENT_ID="<Enter deployment ID here>"
TOOLS_CONTAINER=$(oc get pods |grep $DEPLOYMENT_ID | grep -E "(eventstore)(.*)(tenant-tools)" | awk {'print $1'}) && ORIGIN_IP=`hostname -i | awk '{print $1}'`
After that, you can run the following:
  • For backup
    backup(){ oc exec $TOOLS_CONTAINER -- /eventstore/tools/backup $@; }
  • For restore
    restore(){ oc exec $TOOLS_CONTAINER -- /eventstore/tools/restore $@; }

After this step, you can run all the following backup and restore commands directly on your host machine CLI.

Running backup

The backup operation has three steps: initialization, update and completion. These three steps can be invoked independently or can be invoked in a single command.

Syntax:
backup [COMMAND] [OPTIONS] [arg]

Multiple invocation

  • Initialize the backup:
    backup initialize
    • To overwrite an existing complete backup, use option --backup-name <existing backup name> --force-overwrite
    • An example output:
      Backup storage type: FILESYSTEM
Backup mount point: /eventstore/db/external_db
Backup path: backup
Backup name: 20190613170437
    • Use the backup name shown above as the --backup-name option argument as an example in the following steps.
  • Update (can run zero or multiple times):
    backup update --backup-name <backup name>
  • Complete the backup:
    backup complete --backup-name <backup name>

Single invocation

backup complete --backup-name <backup name>
Note: The backup name must be provided.

An example of COS to COS backup

backup initialize --backup-bucket cosnovasvtbar1      
     
     Backup storage type: COS
     Backup bucket endpoint: s3.us-east.cloud-object-storage.appdomain.cloud
     Backup bucket Name: cosnovasvtbar1
     Backup path: backup
     Backup name: 20190612210149

...
backup update --backup-name 20190612210149 --backup-bucket cosnovasvtbar1
backup complete --backup-name 20190612210149 --backup-bucket cosnovasvtbar1

Example of backup files on COS backup bucket:
2019-06-13 16:59:55          0 backup/20190612210149/BCKP_CMPLT_FLG
2019-06-13 16:55:44  393062400 backup/20190612210149/eventstore_backup_catalog_20190612210149.tar
2019-06-13 16:55:38  158177280 backup/20190612210149/eventstore_backup_compute_0_20190612210149.tar
2019-06-13 16:55:42  158177280 backup/20190612210149/eventstore_backup_compute_1_20190612210149.tar
2019-06-13 16:55:39  158177280 backup/20190612210149/eventstore_backup_compute_2_20190612210149.tar
2019-06-13 16:48:16        544 backup/20190612210149/eventstore_backup_shared_20190612210149/ibm/htap/...
2019-06-13 16:55:40  204032000 backup/20190612210149/eventstore_backup_zookeeper_0_20190612210149.tar
2019-06-13 16:55:41  204083200 backup/20190612210149/eventstore_backup_zookeeper_1_20190612210149.tar
2019-06-13 16:55:40  204001280 backup/20190612210149/eventstore_backup_zookeeper_2_20190612210149.tar
2019-06-13 16:55:23        350 backup/20190612210149/file_meta.txt

An example of filesystem to filesystem backup

backup initialize --backup-name t1 --backup-path bckp_batch1
    
     Backup storage type: FILESYSTEM
     Backup mount point: /eventstore/db/external_db
     Backup path: bckp_batch1
     Backup name: t1

...
backup update --backup-name t1 -backup-path bckp_batch1
backup complete --backup-name t1 -backup-path bckp_batch1

Example of backup files on backup filesystem:
/eventstore/db/external_db/bckp_batch1/t1/BCKP_CMPLT_FLG
/eventstore/db/external_db/bckp_batch1/t1/eventstore_backup_catalog_t1.tar
/eventstore/db/external_db/bckp_batch1/t1/eventstore_backup_shared_t1.tar
/eventstore/db/external_db/bckp_batch1/t1/eventstore_backup_compute_0_t1.tar
/eventstore/db/external_db/bckp_batch1/t1/eventstore_backup_compute_1_t1.tar
/eventstore/db/external_db/bckp_batch1/t1/eventstore_backup_compute_2_t1.tar
/eventstore/db/external_db/bckp_batch1/t1/eventstore_backup_zookeeper_0_t1.tar
/eventstore/db/external_db/bckp_batch1/t1/eventstore_backup_zookeeper_1_t1.tar
/eventstore/db/external_db/bckp_batch1/t1/eventstore_backup_zookeeper_2_t1.tar

Running restore

The restore operation will overwrite the existing database and all the data in it will be lost.

Syntax:
restore [OPTIONS] [arg]

Invocation

restore --backup-name <backup name>

Options

--backup-bucket  Backup bucket name which shares the same endpoint of the bucket used on current installation.
--backup-path    Optional override of the default path ("backup") for the backup on storage.
--backup-name    The name of the backup to restore.
--validate       Only perform pre-restoration validations.

Example of COS to COS restore

restore --backup-name 20190612210149 --backup-bucket cosnovasvtbar1

Example of filesystem to filesystem restore

restore --backup-name t1

Troubleshooting backup and restore

If an error has occurred during backup and restore, you can always look at the logs to find more information. The logs for backup are saved under: 
/ibm/eventstore/engine/shard_db/log/bar/<backup name>
on the host of the tools container, and the format of the log name is:
eventstore_<backup/restore>_<target>_<tools node id>_<backup name>_<time stamp of operation>.log

The logs will show how and what files are being moved during each invocation of backup and restore.

  • Example of logs on disk:
    -rw-r--r--. 1 root root  1082 Jun 13 18:29 eventstore_backup_shared_incremental_2_BACKUP1_20190614012856.log
-rw-r--r--. 1 root root  5158 Jun 13 18:30 eventstore_backup_shared_complete_2_BACKUP1_20190614012856.log
-rw-r--r--. 1 root root   658 Jun 13 18:32 eventstore_backup_zookeeper_0_BACKUP1_20190614012856.log
-rw-r--r--. 1 root root   703 Jun 13 18:32 eventstore_backup_zookeeper_2_BACKUP1_20190614012856.log
-rw-r--r--. 1 root root  3568 Jun 13 18:32 eventstore_backup_catalog_2_BACKUP1_20190614012856.log
-rw-r--r--. 1 root root   658 Jun 13 18:32 eventstore_backup_zookeeper_1_BACKUP1_20190614012856.log
-rw-r--r--. 1 root root 46331 Jun 13 18:45 eventstore_backup_compute_1_BACKUP1_20190614012856.log
-rw-r--r--. 1 root root 46331 Jun 13 18:45 eventstore_backup_compute_2_BACKUP1_20190614012856.log
-rw-r--r--. 1 root root 46331 Jun 13 18:46 eventstore_backup_compute_0_BACKUP1_20190614012856.log
  • Example of log content:
    ...
./db2inst1/NODE0999/EVENTDB/T0000003/
./db2inst1/NODE0999/EVENTDB/T0000003/.SQLCRT.FLG
./db2inst1/NODE0999/EVENTDB/T0000003/C0000000.LRG
./db2inst1/NODE0999/EVENTDB/T0000001/
./db2inst1/NODE0999/EVENTDB/T0000001/.SQLCRT.FLG
./db2inst1/NODE0999/EVENTDB/T0000001/C0000000.TMP/
./db2inst1/NODE0999/EVENTDB/T0000001/C0000000.TMP/SQLTAG.NAM
move: ../../../eventstore/db/ssd/eventstore/backup_temp/BACKUP1/eventstore_backup_catalog_BACKUP1.tar to s3://cosnovasvtbar1/backup/BACKUP1/eventstore_backup_catalog_BACKUP1.tar
Successfully completed backup of catalog path on 2