IBM Software Hub online backup and restore to a different cluster with NetApp Trident protect

1.1 Creating environment variables

Create the following environment variables so that you can copy commands from the documentation and run them without making any changes.

1.2 Configuring NetApp Trident protect for IBM Software Hub

Configure NetApp Trident protect for IBM Software Hub by deploying and running the cpd-trident-protect.py script from a workstation that is connected to the NetApp Trident protect deployment.

1. Log in to Red Hat OpenShift Container Platform as an instance administrator.

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. 1. Set up the Python virtual environment:

      chmod +x ./venv.sh
      ./venv.sh

   2. Activate the Python virtual environment:

      source venv/bin/activate

3. Download and copy the cpd-trident-protect.py script to the current directory.
   Tip: This script is located in the GitHub repository.
4. If you need to update an existing NetApp Trident protect application, you must uninstall the NetApp Trident protect application custom resource by running the following command:

   python cpd-trident-protect.py uninstall \
       --application_name=cpd-operators-tenant \
       --namespace=${PROJECT_CPD_INST_OPERATORS} \
       --trident_protect_operator_ns=trident-protect

5. Install the NetApp Trident protect application and execution hook custom resources by running the following command:

   python cpd-trident-protect.py install \
       --appvault_name=${APPVAULT_NAME} \
       --application_name=cpd-operators-tenant \
       --trident_protect_operator_ns=trident-protect \
       --cpd_operator_ns=${PROJECT_CPD_INST_OPERATORS} \
       --cpdbr_tenant_service_image_prefix="icr.io/cpopen/cpd/cpdbr-oadp"

6. 1. Verify that the NetApp Trident protect application custom resource is in Ready state:

      tridentctl-protect get application -n ${PROJECT_CPD_INST_OPERATORS}

   2. Verify that the NetApp Trident protect execution hook custom resources were created:

      tridentctl-protect get exechook -n ${PROJECT_CPD_INST_OPERATORS}

1.3 Checking the primary instance of every PostgreSQL cluster is in sync with its replicas

The replicas for Cloud Native PostgreSQL and EDB Postgres clusters occasionally get out of sync with the primary node. To check whether this problem exists and to fix the problem, see the troubleshooting topic PostgreSQL cluster replicas get out of sync.

1.4 Removing MongoDB-related ConfigMaps

oc delete cm zen-cs-aux-br-cm -n ${PROJECT_CPD_INST_OPERANDS}

oc delete cm zen-cs-aux-ckpt-cm -n ${PROJECT_CPD_INST_OPERANDS}

oc delete cm zen-cs-aux-qu-cm -n ${PROJECT_CPD_INST_OPERANDS}

oc delete cm zen-cs2-aux-ckpt-cm -n ${PROJECT_CPD_INST_OPERANDS}

1.5 Preparing IBM Knowledge Catalog

If large metadata enrichment jobs are running while an online backup operation is triggered, the Db2 pre-backup hooks might fail because the database cannot be put into a write-suspended state. It is recommended to have minimal enrichment workload while the online backup is scheduled.

1.6 Excluding external volumes from backups

1.7 Checking the status of installed services

2.1 Backing up an IBM Software Hub instance

Create an online backup of each IBM Software Hub instance, or tenant, in your environment by completing the following steps.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you create a backup.

1. Create a backup of the IBM Software Hub instance:

   python cpd-trident-protect.py backup create \
     --backup_name=${BACKUP_NAME} \
     --appvault_name=${APPVAULT_NAME} \
     --application_name=cpd-operators-tenant \
     --namespace=${PROJECT_CPD_INST_OPERATORS} \
     --trident_protect_operator_ns=trident-protect

2. Run the following command to validate the backup is complete. In the following command, the --wait option indicates that the command should wait for the backup process to complete. You can remove the --wait option to check the backup at a single point in time:

   python cpd-trident-protect.py backup status \
     --backup_name=${BACKUP_NAME} \
     --namespace=${PROJECT_CPD_INST_OPERATORS} \
     --trident_protect_operator_ns=trident-protect \
     --wait

2.2 Optional: Deleting unused NetApp Trident protect backups

You can delete any on-demand backups that you created with NetApp Trident protect if you no longer need them. Deleting unused backups can free up the object storage disk and cleanup old backups.

On-demand backups of IBM Software Hub have two parts:

• A NetApp Trident protect volume backup
• A hybrid Velero resource backup

Use the following commands to delete the entire NetApp Trident protect backup:

tridentctl-protect delete backup ${BACKUP_NAME} \
--namespace=${PROJECT_CPD_INST_OPERATORS} \
--tp-namespace=trident-protect

cpd-cli oadp tenant-backup delete ${BACKUP_NAME}

3.1 Preparing the target cluster

Prepare the target cluster that you want to use to restore IBM Software Hub.

1. Make sure that the target cluster meets the following requirements:
   • The target cluster has the same storage classes as the source cluster.
   • For environments that use a private container registry, such as air-gapped environments, the target cluster has the same image content source policy as the source cluster. For details on configuring the image content source policy, see Configuring an image content source policy for IBM Software Hub images.
   • The target cluster must be able to pull software images. For details, see Updating the global image pull secret for IBM Software Hub.
   • The deployment environment of the target cluster is the same as the source cluster.
     • The target cluster uses the same hardware architecture as the source cluster. For example, x86-64.
     • The target cluster is on the same OpenShift version as the source cluster.
     • The target cluster allows for the same node configuration as the source cluster. For example, if the source cluster uses a custom KubeletConfig, the target cluster must allow the same custom KubeletConfig.
     • Moving between IBM Cloud and non-IBM Cloud deployment environments is not supported.
2. If you are using node labels as the method for identifying nodes in the cluster, re-create the labels on the target cluster.
   Best practice: Use node labels instead of node lists when you are restoring a IBM Software Hub deployment to a different cluster, especially if you plan to enforce node pinning. Node labels enable node pinning with minimal disruption. To learn more, see Passing node information to IBM Software Hub.
3. Install NetApp Trident protect.
   Note: Install the same version that you installed on the source cluster.
4. Install OADP.
   Note: Install the same OADP version that you installed on the source cluster. Follow the instructions in 4. Installing IBM Software Hub OADP backup and restore utility components, except for setting up object storage to store backups.
5. Install or upgrade cpdbr service role-based access controls (RBACs).
   Notes:

   • Ensure that the same version of the cpdbr service is installed on the source and target clusters.

   • It is recommended that you install the latest version of the cpdbr service. If you previously installed the service, upgrade the service by doing the upgrade steps.

   • When you install the cpdbr service on the target cluster, a warning that the IBM Software Hub instance (tenant) operator project (namespace), ${PROJECT_CPD_INST_OPERATORS}, doesn't exist appears. You can ignore this warning.

   • When the cpdbr service is installed on a target cluster, only the required permissions and cluster role bindings are created, because the IBM Software Hub projects (namespaces) aren't yet restored.

   Install the cpdbr service RBACs on the target cluster

   1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

      ${OC_LOGIN}

      Remember: OC_LOGIN is an alias for the oc login command.
   2. Configure the client to set the OADP project:

      cpd-cli oadp client config set namespace=${OADP_PROJECT}

   3. In the IBM Software Hub instance operators project, install the cpdbr service.
      Note: Run the cpdbr installation command in the IBM Software Hub operators project even though the project does not yet exist in the target cluster. Do not manually create the project on the target cluster. The project is created during the IBM Software Hub restore process.

      Environments with the scheduling service

      cpd-cli oadp install \
      --component=cpdbr-tenant \
      --namespace=${OADP_PROJECT} \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpd-scheduler-namespace=${PROJECT_SCHEDULING_SERVICE} \
      --log-level=debug \
      --rbac-only=true \
      --verbose

      Environments without the scheduling service

      cpd-cli oadp install \
      --component=cpdbr-tenant \
      --namespace=${OADP_PROJECT} \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --log-level=debug \
      --rbac-only=true \
      --verbose

   4. Verify that the ClusterRole and ClusterRoleBinding were created:

      oc get clusterrole cpdbr-tenant-service-clusterrole

      oc get clusterrolebinding cpdbr-tenant-service-crb

      If the cluster role bindings were created successfully, these commands return output like in the following examples:

      NAME                               CREATED AT
      cpdbr-tenant-service-clusterrole   <timestamp>

      NAME                       ROLE                                           AGE
      cpdbr-tenant-service-crb   ClusterRole/cpdbr-tenant-service-clusterrole   45h

   Upgrade the cpdbr service on the target cluster

   1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

      ${OC_LOGIN}

      Remember: OC_LOGIN is an alias for the oc login command.
   2. Upgrade the cpdbr service.

      Environments with the scheduling service

      cpd-cli oadp install \
      --upgrade=true \
      --component=cpdbr-tenant \
      --namespace=${OADP_PROJECT} \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpd-scheduler-namespace=${PROJECT_SCHEDULING_SERVICE} \
      --log-level=debug \
      --rbac-only=true \
      --verbose

      Environments without the scheduling service

      cpd-cli oadp install \
      --upgrade=true \
      --component=cpdbr-tenant \
      --namespace=${OADP_PROJECT} \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --log-level=debug \
      --rbac-only=true \
      --verbose

6. Install Certificate manager and the IBM License Service.

   For details, see Installing shared cluster components for IBM Software Hub.

   Note: You must install the same version of Certificate manager and the IBM License Service that is installed on the source cluster.
7. If your IBM Software Hub deployment includes the following services, install and set up prerequisite software.

   Instructions for installing prerequisite software are located in Installing prerequisite software.

   Prerequisite software	Service
   GPU operators	An asterisk (*) indicates that the service requires GPU in some situations. IBM Knowledge Catalog Premium * IBM Knowledge Catalog Standard * Watson Machine Learning * Watson Studio Runtimes * watsonx.ai™ watsonx Assistant * Watsonx BI watsonx Code Assistant™ watsonx Code Assistant for Red Hat Ansible® Lightspeed watsonx Code Assistant for Z watsonx Code Assistant for Z Agentic 5.2.1 and later watsonx Code Assistant for Z Code Explanation watsonx Code Assistant for Z Code Generation 5.2.1 and later watsonx.data™ * watsonx.data Premium watsonx.data intelligence watsonx™ Orchestrate *
   Red Hat OpenShift AI	An asterisk (*) indicates that the service requires Red Hat OpenShift AI in some situations. IBM Knowledge Catalog Premium * IBM Knowledge Catalog Standard * watsonx.ai watsonx Assistant * Watsonx BI watsonx Code Assistant watsonx Code Assistant for Red Hat Ansible Lightspeed watsonx Code Assistant for Z watsonx Code Assistant for Z Agentic 5.2.1 and later watsonx Code Assistant for Z Code Explanation watsonx Code Assistant for Z Code Generation 5.2.1 and later watsonx.data Premium watsonx.data intelligence watsonx Orchestrate *
   Multicloud Object Gateway	Watson Discovery Watson Speech services watsonx Assistant watsonx Orchestrate
   Red Hat OpenShift Serverless Knative Eventing	watsonx Assistant watsonx Orchestrate

   Warning: Do not create the secrets that the service needs to communicate with Multicloud Object Gateway. Instead, the secrets must be created as a post-restore task.

3.2 Cleaning up the target cluster after a previous restore

If you previously restored a IBM Software Hub backup or a previous restore attempt was unsuccessful, delete the IBM Software Hub instance projects (namespaces) in the target cluster before you try another restore.

Resources in the IBM Software Hub instance are watched and managed by operators and controllers that run in other projects. To prevent corruption or out of sync operators and resources when you delete a IBM Software Hub instance, Kubernetes resources that have finalizers specified in metadata must be located, and those finalizers must be deleted before you can delete the IBM Software Hub instance.

1. Log in to Red Hat OpenShift Container Platform as an instance administrator.

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Download the cpd-pre-restore-cleanup.sh script from https://github.com/IBM/cpd-cli/tree/master/cpdops/5.2.0.
3. If the tenant operator project exists and has the common-service NamespaceScope custom resource that identifies all the tenant projects, run the following command:

   ./cpd-pre-restore-cleanup.sh --tenant-operator-namespace="${PROJECT_CPD_INST_OPERATORS}"

4. If the tenant operator project does not exist or specific IBM Software Hub projects need to be deleted, run the following command.

   If the common-service NamespaceScope custom resource is not available and additional projects, such as tethered projects, need to be deleted, modify the list of comma-separated projects in the --additional-namespaces option as necessary.

   ./cpd-pre-restore-cleanup.sh --additional-namespaces="${PROJECT_CPD_INST_OPERATORS},${PROJECT_CPD_INST_OPERANDS}"

5. If the IBM Software Hub scheduling service was installed, uninstall it.

   For details, see Uninstalling the scheduling service.

4.1 Restoring the scheduling service

Restore for the scheduling service is not supported. Reinstall it on your target cluster.

4.2 Restoring an IBM Software Hub instance

Restore an IBM Software Hub instance by completing the following steps.

Check the Known issues and limitations for IBM Software Hub page for any workarounds that you might need to do before you restore a backup.

1. Identify the PATH value of the NetApp Trident protect backup that you want to restore:

   tridentctl-protect get appvaultcontent cpd-appvault --show-paths -n trident-protect --app cpd-operators-tenant --show-paths --show-resources backup

2. Restore the IBM Software Hub instance. In the following script, add the PATH information from the previous step for the APP_ARCHIVE_PATH environment variable:

   APP_ARCHIVE_PATH="Use the PATH value from the previous step."
   
   python cpd-trident-protect.py restore create \
     --restore_name=${RESTORE_NAME} \
     --namespace=${PROJECT_CPD_INST_OPERATORS} \
     --path=${APP_ARCHIVE_PATH} \
     --namespace_mappings="${PROJECT_CPD_INST_OPERATORS}:${PROJECT_CPD_INST_OPERATORS},${PROJECT_CPD_INST_OPERANDS}:${PROJECT_CPD_INST_OPERANDS}" \
     --appvault_name=${APPVAULT_NAME} \
     --trident_protect_operator_ns=trident-protect
     --oadp_namespace=${OADP_PROJECT}

   Note: Because of a limitation in NetApp Trident protect, the --namespace_mappings field is required. Each namespace must be specified and mapped to the original value. The IBM Software Hub integration does not support restoring to different namespaces.
3. Run the following command to validate the restore is complete. In the following command, the --wait option indicates that the command should wait for the restore process to complete. You can remove the --wait option to check the restore at a single point in time:

   python cpd-trident-protect.py restore status \
     --restore_name=${RESTORE_NAME} \
     --namespace=${PROJECT_CPD_INST_OPERATORS} \
     --trident_protect_operator_ns=trident-protect \
     --wait