Backup and restore

Edit online

To recover from any data loss that might occur, regularly back up your IBM® Security QRadar® EDR data. You can use the backup and restore process to support a disaster recovery that requires a redeployment of your environment.

When you install QRadar EDR, you configure a suitable storage class in the cluster. You support the configuration with one or more persistent volumes of suitable size. For more information about storage, see Storage requirements.

You provide secure storage for the backups that is mounted as a Persistent Volume Claim (PVC) in a pod. The backup and restore pod contains all of the necessary utilities for the backup and restore process. The backup and restore pod is deployed automatically as part of the installation or upgrade of QRadar EDR. By default the last 10 backups are kept for each data store.

Important: You must install Red Hat OpenShift CLI 4.12 or later before you backup or restore data. See Install Red Hat OpenShift CLI 4.12 or later.

The following table shows where the backup files are stored in the backup and restore pod, and the backup file name convention that is used for each data store.

Table 1. Data store locations and file name conventions
Data store Location Backup file name
Cassandra /opt/data/backup/cassandra cassandra_backup_<YYYY_MM_DD__HH_MM_SS>.gz
Elasticsearch /opt/data/backup/elasticsearch elasticsearch_backup_<YYYY_MM_DD__HH_MM_SS>.gz
QRadar EDR /opt/data/backup/reaqta reaqta_backup_<YYYY_MM_DD__HH_MM_SS>.gz
Postgres /opt/data/backup/pg pg_backup_default_<YYYY_MM_DD__HH_MM_SS>.gz

Install Red Hat OpenShift CLI 4.12 or later

Edit online

The Red Hat OpenShift CLI client helps you develop, build, deploy, and run your applications on any Red Hat OpenShift or Kubernetes cluster. It also includes the administrative commands for managing a cluster under the adm subcommand.

Procedure

  1. Download Red Hat OpenShift CLI 4.12 or later from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable-4.12/. The file to download is called openshift-client-<platform>-<version>.tar.gz.
  2. Extract the binary file that you downloaded by typing the following command, where <oc_cli_archive_file> is the name of the archive file that you downloaded. 
    tar -xf <oc_cli_archive_file>
  3. Modify the permissions of the binary file by typing the following command, where <oc_cli_binary> is the name of the Red Hat OpenShift binary that you extracted from the archive.
  4. Move the binary file to the /usr/local/bin directory by typing the following command. 
    mv <oc_cli_binary> /usr/local/bin/oc
    Tip: If this command returns a No such file or directory or Not a directory error message, create the /usr/local/bin directory by typing the following command.
    sudo mkdir /usr/local/bin
  5. Ensure that the Red Hat OpenShift CLI client is working by typing the following command. 
    oc version
    Tip: MacOS users might see a message that this tool cannot be opened because it is from an unidentified developer. Close this message and go to System Preferences > Security & Privacy. On the General tab, click Open Anyway or Allow Anyway. Repeat the oc version command.

Backing up data

Edit online

Before you begin

You must have cluster administration privileges.

You must have a 16, 24, or 32-byte Advanced Encryption Standard (AES) key. For more information, see Generating an AES key.

Procedure

  1. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
    • Using a username and password.
      oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>
    • Using a token.
      oc login --token=<token> --server=<openshift_url>
  2. Switch to your QRadar EDR namespace by typing the following command. 
    oc project <qradar_edr_namespace>
  3. Ensure that the backup and restore pod is running by typing the following command. 
    oc get pods -l name=cp4s-backup-restore -n <qradar_edr_namespace>

    When the command completes, take note in the output of the full pod name. In the following example output, the full pod name is cp4s-backup-restore-5f59b99495-9rqtc.

    cp4s-backup-restore-5f59b99495-9rqtc 1/1 Running 02d15
  4. Back up QRadar EDR data stores by typing the following command.
    Table 2. Backup parameters
    Parameter Required Description
    -a <aes_key> Yes <aes_key> is your AES key.
    -k <number_of_backups_to_keep> No <number_of_backups_to_keep> is an integer that indicates the number of backups to keep. If you do not specify this parameter, the 10 most recent backups are kept. 
    oc exec <backup_and_restore_pod> -- /opt/bin/backup-cp4s -a <aes_key> -k <number_of_backups_to_keep>

    For example, to back up all data stores type the following command.

    oc exec <backup_and_restore_pod> -- /opt/bin/backup-cp4s -a <aes_key>

    For example, to back up all data stores and keep the last five backups type the following command.

    oc exec <backup_and_restore_pod> -- /opt/bin/backup-cp4s -a <aes_key> -k 5

    The backup process can take a long time to complete when the data stores are large. The Cassandra data store is often large.

  5. Optional: If you want to determine the size of your backups before you copy them outside of your cluster, type the following command. 
    oc exec <backup_and_restore_pod> -- du -sh /opt/data/backup
  6. Optional: It is good practice to maintain a copy of the backup files in a secure alternative location outside of the backup files to another system outside of your cluster. If you want to copy your backup files to your local system, type one of the following commands.
    • To copy the whole /opt/data/backup folder, type the following command.
      rsync --rsh='oc rsh' -av -c --inplace --partial --append --progress <backup_and_restore_pod>:/opt/data/backup <local_path>

      The result is a copy of the whole /opt/data/backup folder on your local system, in your current directory, in <local_path>/backup. This folder can then be transferred to an appropriate secure storage location.

    • To copy an individual backup file, type the following command.
      rsync --rsh='oc rsh' -av -c --inplace --partial --append --progress <backup_and_restore_pod>:/opt/data/backup/<path_to>/<file_name> <local_path>/<file_name>

      The result is a copy of the file on your local system, in your current directory, in <local_path>/backup. This file can then be transferred to an appropriate secure storage location.

    Tip: If the rsync command times out, run it again to continue copying files.

Scheduling backup

Edit online

QRadar EDR provides a support action to schedule data backup.

Before you begin

To access the schedule_cp4s_full_backup action, you must install the command-line interface (CLI) utility cpctl from the cp-serviceability pod. For more information, see Installing the cpctl utility to access support actions - update link.

About this task

The schedule_cp4s_full_backup action runs a Red Hat OpenShift Container Platform cron job that creates a backup regularly, according to your schedule.

The schedule_cp4s_full_backup action uses the following parameters.
Parameter Default Required Description
--password none Yes A user-defined password that is used to encrypt the backup files, it is required during the restore process. This password cannot be recovered if it is lost.
--token none Yes A token that is generated by the administrator by running oc whoami -t on the local system.
--schedule ("0 0 * * */6") No The cron job schedule that is used to set the schedule for the backup.
--airgap none No The local registry URL that was used to deploy Cloud Pak for Security.
--disable false No To disable (false) or enable (true) the scheduled backup.
--keepfiles 7 No To configure the backup file maintenance process. When the number of backup files that are maintained is greater than the value of this parameter, the oldest backup files are deleted.

Procedure

  1. To set up the schedule, type the following command.
    • In an online environment 
      cpctl tools schedule_cp4s_full_backup --token "$(oc whoami -t)" --password <add-any-password-of-your-choice> --schedule <your_schedule>
    • In an air-gapped environment 
      cpctl tools schedule_cp4s_full_backup --token "$(oc whoami -t)" --password <add-any-password-of-your-choice> --schedule <your_schedule> --airgap <local-docker-registry:5000>
  2. It is good practice to maintain a copy of the backup files in a secure alternative location outside of the backup files to another system outside of your cluster. If you want to copy your backup files to your local system, type one of the following commands.
    • To copy the whole /opt/data/backup folder, type the following command.
      rsync --rsh='oc rsh' -av -c --inplace --partial --append --progress <backup_and_restore_pod>:/opt/data/backup <local_path>

      The result is a copy of the whole /opt/data/backup folder on your local system, in your current directory, in <local_path>/backup. This folder can then be transferred to an appropriate secure storage location.

    • To copy an individual backup file, type the following command.
      rsync --rsh='oc rsh' -av -c --inplace --partial --append --progress <backup_and_restore_pod>:/opt/data/backup/<path_to>/<file_name> <local_path>/<file_name>

      The result is a copy of the file on your local system, in your current directory, in <local_path>/backup. This file can then be transferred to an appropriate secure storage location.

    Tip: If the rsync command times out, run it again to continue copying files.
  3. If you need to disable the scheduled backup, type the following command. 
    cpctl tools schedule_cp4s_full_backup --token "$(oc whoami -t)" --password <add-any-password-of-your-choice> --disable

Restoring data

Edit online

When the restore process is completed, data is restored and the system returns to the state at the time of the backup.

Before you begin

You must have cluster administration privileges.

Because the Cassandra data store can be large, you might need to increase the memory used by the sstableloader process, or configure Cassandra to use the nodetool process.

sstableloader
If you are restoring Cassandara data to a different Red Hat OpenShift cluster, or to the same cluster but with a different number of Cassandra nodes, you must use sstableloader. Depending on the size of the backup, you might need to increase the memory allocated to sstableloader before you restore Cassandra data. For more information, see Increasing the memory for Cassandra restoration.
nodetool
If you are restoring Cassandra data to the same cluster and with the same number of Cassandra nodes, use nodetool. For more information, see Configuring Cassandra restoration to use Nodetool
Important: You must install Red Hat OpenShift CLI 4.12 or later before you backup or restore data. See Install Red Hat OpenShift CLI 4.12 or later.

Procedure

  1. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
    • Using a username and password.
      oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>
    • Using a token.
      oc login --token=<token> --server=<openshift_url>
  2. Switch to your QRadar EDR namespace by typing the following command. 
    oc project <qradar_edr_namespace>
  3. Ensure that the backup and restore pod is running by typing the following command. 
    oc get pods -l name=cp4s-backup-restore -n <qradar_edr_namespace>

    When the command completes, take note in the output of the full pod name. In the following example output, the full pod name is cp4s-backup-restore-5f59b99495-9rqtc.

    cp4s-backup-restore-5f59b99495-9rqtc 1/1 Running 02d15
  4. If your data is stored outside of your cluster, copy the backup files to your cluster by typing the following command. 
    rsync --rsh='oc rsh' -av -c --inplace --partial --append --progress <local_path>/ <backup_and_restore_pod>:/opt/data/backup
  5. Stop the event-hive pod by typing the following command. 
    oc scale deployments/reaqta-hive-event-hive --replicas=0 -n <qradar_edr_namespace>
  6. Restore QRadar EDR data stores by typing the following command.
    Table 3. Backup parameters
    Parameter Required Description
    -a <aes_key> Yes <aes_key> is your AES key.
    -f <file_name> No <file_name> is a specific backup file name.
    Important: Always restore Cassandra and Elasticsearch from the same backup at the same time to prevent synchronization issues.
    		 
    oc exec <backup_and_restore_pod> -- /opt/bin/restore-cp4s -a <aes_key> -f <file_name>

    For example, to restore all data stores type the following command.

    oc exec <backup_and_restore_pod> -- /opt/bin/restore-cp4s -a <aes_key>

    For example, to restore a specific version of the QRadar EDR data store type the following command.

    oc exec <backup_and_restore_pod> -- /opt/bin/restore-cp4s -a <aes_key> -f reaqta_backup_<YYYY_MM_DD__HH_MM_SS>.gz
  7. Start the event-hive pod by typing the following command. 
    oc scale deployments/reaqta-hive-event-hive --replicas=1 -n <qradar_edr_namespace>

Results

After the restore is complete, allow up to 15 minutes for pods to complete the restart operations. If the first attempt at restoring the system is not successful for any reason, the full restore procedure can be run again without any impact.

What to do next

Log in to your QRadar EDR Dashboard and verify that you see the restored data.

Increasing the memory for Cassandra restoration

Edit online

Procedure

  1. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
    • Using a username and password.
      oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>
    • Using a token.
      oc login --token=<token> --server=<openshift_url>
  2. Confirm the current configuration parameters in your deployment by typing the following command. 
    oc get ISCInventory iscplatform -o yaml
  3. Create a .yaml file that contains the following text. 
    apiVersion: isc.ibm.com/v1
kind: CP4SConfiguration
metadata:
  name: cassandra-restore-tuning
  namespace: <qradar_edr_namespace>
spec:
  values:
    - featureFlags.reaqta.hive.cassandra.restore.heap_size=250M
    - featureFlags.reaqta.hive.cassandra.restore.max_heap_size=1G
  4. Apply the configuration by typing the following command. 
    oc apply -f <file_name>
  5. Verify that the configuration was applied successfully by typing the following command. 
    oc get cassandra reaqta-hive-cassandra -o yaml

What to do next

If you need to restore the Cassandra memory to the default, delete the .yaml file by typing the following command.
oc delete -f <file_name>
After you delete the .yaml file, verify that the backup and restore pod is running by typing the following command.
oc get pods -l name=reaqta-hive-event-hive -n <qradar_edr_namespace>

Configuring Cassandra restoration to use Nodetool

Edit online

Procedure

  1. Log in to your Red Hat OpenShift Container Platform cluster as a cluster administrator by typing one of the following commands, where <openshift_url> is the URL for your Red Hat OpenShift Container Platform environment.
    • Using a username and password.
      oc login <openshift_url> -u <cluster_admin_user> -p <cluster_admin_password>
    • Using a token.
      oc login --token=<token> --server=<openshift_url>
  2. Confirm the current configuration parameters in your deployment by typing the following command. 
    oc get ISCInventory iscplatform -o yaml
  3. Create a .yaml file that contains the following text. 
    apiVersion: isc.ibm.com/v1
kind: CP4SConfiguration
metadata:
  name: cassandra-restore-mode
  namespace: <qradar_edr_namespace>
spec:
  values:
    - featureFlags.reaqta.hive.cassandra.restore.mode=nodetool
  4. Apply the configuration by typing the following command. 
    oc apply -f <file_name>
  5. Verify that the configuration was applied successfully by typing the following command. 
    oc get cassandra reaqta-hive-cassandra -o yaml

What to do next

If you need to set Cassandra restoration back to sstableloader, delete the .yaml file by typing the following command.
oc delete -f <file_name>
After you delete the .yaml file, verify that the backup and restore pod is running by typing the following command.
oc get pods -l name=reaqta-hive-event-hive -n <qradar_edr_namespace>