Restoring database data in a geo-redundant hybrid deployment

You can restore existing topology data for the hybrid installation, if backed up earlier. The backing up of data is helpful during your system update, or during maintenance.

kubectl get secret noi-cassandra-auth-secret -o yaml

Preparing your system for data restoration

1. Authenticate into the Agile Service Manager Kubernetes namespace on the primary data center.
2. Deploy the kPodLoop bash shell function.
   kPodLoop is a bash shell function that allows a command to be run against matching Kubernetes containers. You can copy it into the shell.

   kPodLoop() {
    __podPattern=$1
    __podCommand=$2
    __podList=$( kubectl get pods --field-selector=status.phase=Running --no-headers=true --output=custom-columns=NAME:.metadata.name | grep ${__podPattern} )
    printf "Pods found: $(echo -n ${__podList})\n"
    for pod in ${__podList}; do
       container=`oc get pods ${pod} -o jsonpath='{.spec.containers[*].name}' | tr ' ' '\n' | head -1`
       printf "\n===== EXECUTING COMMAND in pod: %-42s =====\n" ${pod}
       oc exec -c ${container} ${pod} -- bash -c "${__podCommand}"
       printf '_%.0s' {1..80}
       printf "\n"
    done;
   }

   kPodLoop bash shell function filters the pods to run the commands against those pods only that are in a 'Running' phase. This filter ensures that the configuration pods that are only running as part of your installation, like the secret generator pod, are skipped.
3. Make a note of the scaling of Agile Service Manager pods.

   oc get pods --field-selector=status.phase=Running --no-headers=true --output=custom-columns=CNAME:.metadata.ownerReferences[0].name | grep primary-topology | uniq --count

   Example output:

   1 primary-topology-dns-observer
   1 primary-topology-docker-observer
   3 primary-topology-elasticsearch
   1 primary-topology-file-observer
   1 primary-topology-kubernetes-observer
   1 primary-topology-layout
   1 primary-topology-merge
   1 primary-topology-noi-gateway
   1 primary-topology-noi-probe
   1 primary-topology-observer-service
   1 primary-topology-search
   1 primary-topology-status
   1 primary-topology-topology
   1 primary-topology-ui-api

4. Verify access to each Cassandra database (this command returns a list of keyspaces from each Cassandra node).

   kPodLoop primary-cassandra "CASSANDRA_USER=\$(cat \$CASSANDRA_AUTH_USERNAME_FILE); CASSANDRA_PASS=\$(cat \$CASSANDRA_AUTH_PASSWORD_FILE); cqlsh -u \${CASSANDRA_USER} -p \${CASSANDRA_PASS} \${CASSANDRA_CLIENT_ENCRYPTION_ENABLED:+--ssl} -e \"DESC KEYSPACES;\""
   

5. Scale down Agile Service Manager pods.

   oc scale deployment --replicas=0 primary-topology-dns-observer
   oc scale deployment --replicas=0 primary-topology-file-observer
   oc scale deployment --replicas=0 primary-topology-kubernetes-observer
   oc scale deployment --replicas=0 primary-topology-observer-service
   oc scale deployment --replicas=0 primary-topology-noi-gateway
   oc scale deployment --replicas=0 primary-topology-noi-probe
   oc scale deployment --replicas=0 primary-topology-layout
   oc scale deployment --replicas=0 primary-topology-merge
   oc scale deployment --replicas=0 primary-topology-status
   oc scale deployment --replicas=0 primary-topology-search
   oc scale deployment --replicas=0 primary-topology-ui-api
   oc scale deployment --replicas=0 primary-topology-topology

   The Cassandra pods are left active. Cassandra pods need to be kept running in order to execute the backup of their data.

   Important: Include in this scale down any additional observers that you installed in your deployment.
6. Verify that scaling down was successful.

   oc get pods --field-selector=status.phase=Running  | grep primary-topology

   The Agile Service Manager services have now been scaled down, and the Cassandra database contents will not be modified anymore.
7. Repeat steps 1 - 6 for the backup data center.

Restore data

8. On the primary data center, update the Cassandra restore script to suppress the truncation of restored data and the copying of data to the backup date center.
   Note: The restore_cassandra.sh tool truncates all the data in the target table each time it is used, and despite the restore being targeted at one Cassandra node only, the truncate is propagated to all nodes. In order to suppress the truncate step, you must update the restore script on all the nodes except the first node.
   1. Copy cassandra_functions.sh out of one of the asm-cassandra nodes.

      oc cp primary-cassandra-0-0:/opt/ibm/backup_scripts/cassandra_functions.sh /tmp/cassandra_functions.sh

      Edit cassandra_functions.sh

      vi /tmp/cassandra_functions.sh

   2. Locate the call to truncate_all_tables within the restore() function and comment out the appropriate lines, as in the following example:

      Printf "`date` Starting Restore \n"
      
      #### truncate_all_tables
      #### testResult $? "truncate tables"

   3. Save the file. Then, copy the file back to all nodes, except the first Cassandra node.

      oc cp /tmp/cassandra_functions.sh primary-cassandra-1-0:/opt/ibm/backup_scripts/cassandra_functions.sh

      oc cp /tmp/cassandra_functions.sh primary-cassandra-2-0:/opt/ibm/backup_scripts/cassandra_functions.sh

9. Locate the timestamps of restoring the backups from each Cassandra node.
   Each node's backup was started at a similar time, so the timestamps may differ by a few seconds. In the following example a backup was performed at about 2019-06-11 09:36, and grep is then used to filter to these backup archives.

   Tip: You can ignore this step if you are about to apply the most recent backup. If you do, the -t parameter can be omitted during all subsequent steps.

   kPodLoop primary-cassandra "ls -larth \${CASSANDRA_DATA}/../backup_tar | grep 2019-06-11-09"
   
   Pods found: primary-cassandra-0-0 primary-cassandra-1-0 primary-cassandra-2-0
   
   ===== EXECUTING COMMAND in pod: primary-cassandra-0-0                            =====
   -rwxrwxr-x 1 cassandra cassandra 524M Jun 11 09:37 cassandra_primary-topology-cassandra-0-0_KS_system_schema_KS_system_KS_system_distributed_KS_system_auth_KS_janusgraph_KS_system_traces_date_2019-06-11-0936-04.tar
   ________________________________________________________________________________
   
   ===== EXECUTING COMMAND in pod: primary-cassandra-1-0                            =====
   -rwxrwxr-x 1 cassandra cassandra 565M Jun 11 09:37 cassandra_primary-topology-cassandra-1-0_KS_system_schema_KS_system_KS_system_distributed_KS_system_auth_KS_janusgraph_KS_system_traces_date_2019-06-11-0936-07.tar
   ________________________________________________________________________________
   
   ===== EXECUTING COMMAND in pod: primary-cassandra-2-0                            =====
   -rwxrwxr-x 1 cassandra cassandra 567M Jun 11 09:37 cassandra_primary-topology-cassandra-2-0_KS_system_schema_KS_system_KS_system_distributed_KS_system_auth_KS_janusgraph_KS_system_traces_date_2019-06-11-0936-07.tar
   ________________________________________________________________________________

10. Working across each Cassandra node in the primary data center, restore the relevant backup of the janusgraph keyspace.
    Note: For information about the system_auth keyspace see the Secrets and system_auth keyspace note.

    The specific node pod names can vary depending on your installation and release names used
    1. primary-cassandra-0-0
       Remember: This will cause the existing data in the janusgraph keyspace tables to be truncated.

       kPodLoop primary-cassandra-0-0 "CASSANDRA_USER=\$(cat \$CASSANDRA_AUTH_USERNAME_FILE); CASSANDRA_PASS=\$(cat \$CASSANDRA_AUTH_PASSWORD_FILE); /opt/ibm/backup_scripts/restore_cassandra.sh -k janusgraph  -t 2019-06-11-0936-04 -u \${CASSANDRA_USER} -p \${CASSANDRA_PASS} -f"
       

    2. primary-cassandra-1-0

       kPodLoop primary-cassandra-1-0 "CASSANDRA_USER=\$(cat \$CASSANDRA_AUTH_USERNAME_FILE); CASSANDRA_PASS=\$(cat \$CASSANDRA_AUTH_PASSWORD_FILE); /opt/ibm/backup_scripts/restore_cassandra.sh -k janusgraph  -t 2019-06-11-0936-04 -u \${CASSANDRA_USER} -p \${CASSANDRA_PASS} -f"
       

    3. primary-cassandra-2-0

       kPodLoop primary-cassandra-2-0 "CASSANDRA_USER=\$(cat \$CASSANDRA_AUTH_USERNAME_FILE); CASSANDRA_PASS=\$(cat \$CASSANDRA_AUTH_PASSWORD_FILE); /opt/ibm/backup_scripts/restore_cassandra.sh -k janusgraph  -t 2019-06-11-0936-04 -u \${CASSANDRA_USER} -p \${CASSANDRA_PASS} -f"
       

11. For one node in the primary data center, repair the janusgraph key space.

    kPodLoop primary-cassandra-0 "nodetool repair --full --local janusgraph"

12. For each node in the backup data center, rebuild the Cassandra node from the primary data center.
    1. backup-cassandra-0-0

       kPodLoop backup-cassandra-0-0 "nodetool rebuild -- dc-1"

    2. backup-cassandra-1-0

       kPodLoop backup-cassandra-1-0 "nodetool rebuild -- dc-1"

    3. backup-cassandra-2-0

       kPodLoop backup-cassandra-2-0 "nodetool rebuild -- dc-1"

13. After you run the rebuild scripts, clear historical alert data by using the statusClear crawler from the primary data center.

    https://master_fqdn/1.0/topology/swagger#!/Crawlers/statusClear

Restore services

14. Scale up the services to the original level.
    The original level was obtained in a previous step.

    oc scale deployment --replicas=1 primary-topology-topology
        oc scale deployment --replicas=1 primary-topology-layout
        oc scale deployment --replicas=1 primary-topology-merge
        oc scale deployment --replicas=1 primary-topology-status
        oc scale deployment --replicas=1 primary-topology-search
        oc scale deployment --replicas=1 primary-topology-observer-service
        oc scale deployment --replicas=1 primary-topology-noi-gateway
        oc scale deployment --replicas=1 primary-topology-noi-probe
        oc scale deployment --replicas=1 primary-topology-ui-api
        oc scale deployment --replicas=1 primary-topology-dns-observer
        oc scale deployment --replicas=1 primary-topology-file-observer
        oc scale deployment --replicas=1 primary-topology-rest-observer
        oc scale deployment --replicas=1 primary-topology-kubernetes-observer

15. Scale up the services in the backup data center to the original level.
16. Rebroadcast data to inventory.
    If data in the inventory service is out of sync with data in the Cassandra database, resynchronize it by calling the rebroadcast API of the topology service. This triggers the rebroadcast of all known resources on Kafka, and the inventory service will then index those resources in PostgreSQL. Call the rebroadcast API of the Topology service, by specifying a tenantId.

    https://master_fqdn/1.0/topology/swagger#!/Crawlers/rebroadcastTopology

    For more information, see Backing up database data (OCP).