IBM Software Hub online backup and restore to a different cluster with IBM Fusion

1.1 Creating environment variables

Create the following environment variables so that you can copy commands from the documentation and run them without making any changes.

1.2 Preparing IBM Fusion

Prepare IBM Fusion by setting up one of the clusters as the IBM Fusion backup and restore hub.

1. In IBM Fusion, open the Services page and click the Backup & Restore tile.
2. In the Install service window, select the storage class (RWO) that you want to use to deploy the service.

   The ibm-backup-restore project (namespace) is created on the cluster, and the service is installed in that project.

3. Verify that the hub is in a healthy state by clicking Backup & restore > Topology and checking the Service status column of the hub.

1.3 Checking the contents of the IBM Fusion application for the IBM Software Hub operator

Check that the IBM Fusion application custom resource for the IBM Software Hub operator includes the following information:

• All projects (namespaces) that are members of the IBM Software Hub instance, including:
  • The IBM Software Hub operators project (${PROJECT_CPD_INST_OPERATORS}).
  • The IBM Software Hub operands project (${PROJECT_CPD_INST_OPERANDS}).
  • All tethered projects, if they exist.
• The PARENT_NAMESPACE variable, which is set to ${PROJECT_CPD_INST_OPERATORS}.

Do the following steps:

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. To get the list of all projects that are members of the IBM Software Hub instance, run the following command:

   oc get -n ${PROJECT_FUSION} applications.application.isf.ibm.com ${PROJECT_CPD_INST_OPERATORS} -o jsonpath={'.spec.includedNamespaces'}

3. To get the PARENT_NAMESPACE variable, run the following command:

   oc get -n ${PROJECT_FUSION} applications.application.isf.ibm.com ${PROJECT_CPD_INST_OPERATORS} -o jsonpath={'.spec.variables'}

1.4 Checking the version of cpdbr resources

You must install the correct version of cpdbr resources for the IBM Software Hub version that you are using. For example, if you upgraded IBM Software Hub from version 4.8.4 to 5.2.0, you must also upgrade the cpdbr service to version 5.2.0.

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Check the version of cpdbr-oadp by running the following command:

   oc get po -l component=cpdbr-tenant,icpdsupport/app=br-service -n ${PROJECT_CPD_INST_OPERATORS} -o jsonpath='{.items[0].spec.containers[0].image}'

   Example output:

   icr.io/cpopen/cpd/cpdbr-oadp:5.2.2

3. Check the version of the IBM Fusion backup and restore recipe for IBM Software Hub by running the following command:

   oc get -n ${PROJECT_CPD_INST_OPERATORS} frcpe ibmcpd-tenant -o jsonpath={'.metadata.labels.icpdsupport/version'}

4. Ensure that the values returned by the preceding commands match.

1.5 Expanding PVCs that are smaller than 5Gi when using IBM Storage Scale Container Native storage

If your IBM Software Hub deployment is using IBM Storage Scale Container Native or IBM Fusion Global Data Platform storage, expand Persistent Volume Claims (PVCs) that are smaller than 5Gi to at least that amount to ensure that restoring a backup is successful. For details on expanding PVCs, see Volume Expansion in the IBM Storage Scale Container Storage Interface Driver documentation.

1.6 Checking the primary instance of every PostgreSQL cluster is in sync with its replicas

The replicas for Cloud Native PostgreSQL and EDB Postgres clusters occasionally get out of sync with the primary node. To check whether this problem exists and to fix the problem, see the troubleshooting topic PostgreSQL cluster replicas get out of sync.

1.7 Removing MongoDB-related ConfigMaps

oc delete cm zen-cs-aux-br-cm

oc delete cm zen-cs-aux-ckpt-cm

oc delete cm zen-cs-aux-qu-cm

oc delete cm zen-cs2-aux-ckpt-cm

1.8 Preparing IBM Knowledge Catalog

If large metadata enrichment jobs are running while an online backup operation is triggered, the Db2 pre-backup hooks might fail because the database cannot be put into a write-suspended state. It is recommended to have minimal enrichment workload while the online backup is scheduled.

1.9 Checking the status of installed services

3.1 Preparing the target cluster

Prepare the target cluster that you want to use to restore IBM Software Hub.

1. Make sure that the target cluster meets the following requirements:
   • The target cluster has the same storage classes as the source cluster.
   • For environments that use a private container registry, such as air-gapped environments, the target cluster has the same image content source policy as the source cluster. For details on configuring the image content source policy, see Configuring an image content source policy for IBM Software Hub images.
   • The target cluster must be able to pull software images. For details, see Updating the global image pull secret for IBM Software Hub.
   • The deployment environment of the target cluster is the same as the source cluster.
     • The target cluster uses the same hardware architecture as the source cluster. For example, x86-64.
     • The target cluster is on the same OpenShift version as the source cluster.
     • The target cluster allows for the same node configuration as the source cluster. For example, if the source cluster uses a custom KubeletConfig, the target cluster must allow the same custom KubeletConfig.
     • Moving between IBM Cloud and non-IBM Cloud deployment environments is not supported.
2. If you are using node labels as the method for identifying nodes in the cluster, re-create the labels on the target cluster.
   Best practice: Use node labels instead of node lists when you are restoring a IBM Software Hub deployment to a different cluster, especially if you plan to enforce node pinning. Node labels enable node pinning with minimal disruption. To learn more, see Passing node information to IBM Software Hub.
3. Install one of the following IBM Fusion versions:
   • Version 2.10.x with the latest hotfix
4. In IBM Fusion on the hub (source) cluster, go to Backup & restore > Overview.
5. Under 1. Connect your clusters (optional), click Use Fusion UI and then click Copy snippet.

   The hub connection snippet is copied.

6. In IBM Fusion on the spoke (target) cluster, click Services, and in the Services page, click Backup & Restore Agent.
7. In the Install Service page, paste the hub connection snippet that you previously copied from the hub cluster.
8. Select the storage class (RWO) that you want to use and click Install.

   The ibm-backup-restore project (namespace) is created on the cluster, and the service is installed in that project.

9. Install or upgrade cpdbr service role-based access controls (RBACs).
   Notes:

   • Ensure that the same version of the cpdbr service is installed on the source and target clusters.

   • It is recommended that you install the latest version of the cpdbr service. If you previously installed the service, upgrade the service by doing the upgrade steps.

   • When the cpdbr service is installed on a target cluster, only the required permissions and cluster role bindings are created, because the IBM Software Hub projects (namespaces) aren't yet restored.

   Install the cpdbr service RBACs on the target cluster

   1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

      ${OC_LOGIN}

      Remember: OC_LOGIN is an alias for the oc login command.
   2. Install the cpdbr service.
      Note: Run the cpdbr installation command in the IBM Software Hub operators project even though the project does not yet exist in the target cluster. Do not manually create the project on the target cluster. The project is created during the IBM Software Hub restore process.

      Environments with the scheduling service

      cpd-cli oadp install \
      --component=cpdbr-tenant \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpd-scheduler-namespace=${PROJECT_SCHEDULING_SERVICE} \
      --namespace=${OADP_PROJECT} \
      --log-level=debug \
      --rbac-only=true \
      --verbose

      Environments without the scheduling service

      cpd-cli oadp install \
      --component=cpdbr-tenant \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --namespace=${OADP_PROJECT} \
      --log-level=debug \
      --rbac-only=true \
      --verbose

   3. Verify that the ClusterRole and ClusterRoleBinding were created:

      oc get clusterrole cpdbr-tenant-service-clusterrole

      oc get clusterrolebinding cpdbr-tenant-service-crb

      If the cluster role bindings were created successfully, these commands return output like in the following examples:

      NAME                               CREATED AT
      cpdbr-tenant-service-clusterrole   <timestamp>

      NAME                       ROLE                                           AGE
      cpdbr-tenant-service-crb   ClusterRole/cpdbr-tenant-service-clusterrole   45h

   Upgrade the cpdbr service on the target cluster

   1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

      ${OC_LOGIN}

      Remember: OC_LOGIN is an alias for the oc login command.
   2. Upgrade the cpdbr service.

      Environments with the scheduling service

      cpd-cli oadp install \
      --upgrade=true \
      --component=cpdbr-tenant \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --cpd-scheduler-namespace=${PROJECT_SCHEDULING_SERVICE} \
      --namespace=${OADP_PROJECT} \
      --rbac-only=true \
      --log-level=debug \
      --verbose

      Environments without the scheduling service

      cpd-cli oadp install \
      --upgrade=true \
      --component=cpdbr-tenant \
      --tenant-operator-namespace=${PROJECT_CPD_INST_OPERATORS} \
      --namespace=${OADP_PROJECT} \
      --rbac-only=true \
      --log-level=debug \
      --verbose

10. Verify the hub and spoke topology.
    1. In IBM Fusion on the hub cluster, go to Backup & restore > Topology.
    2. Verify that the hub and spoke are connected and in a healthy service state.
11. Install Certificate manager and the IBM License Service.

    For details, see Installing shared cluster components for IBM Software Hub.

    Note: You must install the same version of Certificate manager and the IBM License Service that is installed on the source cluster.
12. If your IBM Software Hub deployment includes the following services, install and set up prerequisite software.

    Instructions for installing prerequisite software are located in Installing prerequisite software.

    Prerequisite software	Service
    GPU operators	An asterisk (*) indicates that the service requires GPU in some situations. IBM Knowledge Catalog Premium * IBM Knowledge Catalog Standard * Watson Machine Learning * Watson Studio Runtimes * watsonx.ai™ watsonx Assistant * Watsonx BI watsonx Code Assistant™ watsonx Code Assistant for Red Hat Ansible® Lightspeed watsonx Code Assistant for Z watsonx Code Assistant for Z Agentic 5.2.1 and later watsonx Code Assistant for Z Code Explanation watsonx Code Assistant for Z Code Generation 5.2.1 and later watsonx.data™ * watsonx.data Premium watsonx.data intelligence watsonx™ Orchestrate *
    Red Hat OpenShift AI	An asterisk (*) indicates that the service requires Red Hat OpenShift AI in some situations. IBM Knowledge Catalog Premium * IBM Knowledge Catalog Standard * watsonx.ai watsonx Assistant * Watsonx BI watsonx Code Assistant watsonx Code Assistant for Red Hat Ansible Lightspeed watsonx Code Assistant for Z watsonx Code Assistant for Z Agentic 5.2.1 and later watsonx Code Assistant for Z Code Explanation watsonx Code Assistant for Z Code Generation 5.2.1 and later watsonx.data Premium watsonx.data intelligence watsonx Orchestrate *
    Multicloud Object Gateway	Watson Discovery Watson Speech services watsonx Assistant watsonx Orchestrate
    Red Hat OpenShift Serverless Knative Eventing	watsonx Assistant watsonx Orchestrate

    Warning: Do not create the secrets that the service needs to communicate with Multicloud Object Gateway. Instead, the secrets must be created as a post-restore task.

3.2 Cleaning up the target cluster after a previous restore

If you previously restored a IBM Software Hub backup or a previous restore attempt was unsuccessful, delete the IBM Software Hub instance projects (namespaces) in the target cluster before you try another restore.

Resources in the IBM Software Hub instance are watched and managed by operators and controllers that run in other projects. To prevent corruption or out of sync operators and resources when you delete a IBM Software Hub instance, Kubernetes resources that have finalizers specified in metadata must be located, and those finalizers must be deleted before you can delete the IBM Software Hub instance.

1. Log in to Red Hat OpenShift Container Platform as an instance administrator.

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Download the cpd-pre-restore-cleanup.sh script from https://github.com/IBM/cpd-cli/tree/master/cpdops/5.2.0.
3. If the tenant operator project exists and has the common-service NamespaceScope custom resource that identifies all the tenant projects, run the following command:

   ./cpd-pre-restore-cleanup.sh --tenant-operator-namespace="${PROJECT_CPD_INST_OPERATORS}"

4. If the tenant operator project does not exist or specific IBM Software Hub projects need to be deleted, run the following command.

   If the common-service NamespaceScope custom resource is not available and additional projects, such as tethered projects, need to be deleted, modify the list of comma-separated projects in the --additional-namespaces option as necessary.

   ./cpd-pre-restore-cleanup.sh --additional-namespaces="${PROJECT_CPD_INST_OPERATORS},${PROJECT_CPD_INST_OPERANDS}"

5. If the IBM Software Hub scheduling service was installed, uninstall it.

   For details, see Uninstalling the scheduling service.

3.3 Validating cpdbr service permissions

After the cpdbr service is installed on the target cluster, check that the proper permissions that the service needs exist.

1. Check roles and rolebindings that are needed for the IBM Software Hub restore:

   oc get clusterrole cpdbr-tenant-service-clusterrole

   oc get clusterrolebinding cpdbr-tenant-service-crb

2. Check that the IBM Software Hub operators service account is associated with the cpdbr-tenant-service-role role in the kube-public project (namespace):

   oc describe -n kube-public rolebinding cpdbr-tenant-service-rb | grep ${PROJECT_CPD_INST_OPERATORS}

   Example output:

   Name:         cpdbr-tenant-service-rb
   Labels:       component=cpdbr-tenant
                 icpdsupport/addOnId=cpdbr
                 icpdsupport/app=br-service
                 icpdsupport/cpdbr=true
   Annotations:  <none>
   Role:
     Kind:  Role
     Name:  cpdbr-tenant-service-role
   Subjects:
     Kind            Name                     Namespace
     ----            ----                     ---------
     ServiceAccount  cpdbr-tenant-service-sa  cpd-op
     ServiceAccount  cpdbr-tenant-service-sa  group1-op
     ServiceAccount  cpdbr-tenant-service-sa  cpd-op-grp2

5.1 Passing node information and applying cluster HTTP proxy settings or other RSI patches to thecontrol plane

If you use node lists to pin pods to nodes, you must re-run the cpd-cli manage apply-entitlement command after you restore IBM Software Hub on the target cluster. Any pods that need to be rescheduled will be unavailable while they are moved to different nodes. For more information, see Passing node information to IBM Software Hub.

cpd-cli manage apply-rsi-patches --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} -vvv

5.2 Patching Cognos Analytics instances

5.3 Patching Watson OpenScale database

5.4 Resetting status of ongoing Watson OpenScale model evaluations

When a Watson OpenScale instance is restored, some of its features, such as scheduled or on-demand model evaluations, might not function properly. You must reset the status of ongoing model evaluations. For details, see Resetting status of ongoing Watson OpenScale model evaluations.

5.5 Restarting IBM Knowledge Catalog metadata import jobs

5.6 Restarting IBM Knowledge Catalog metadata enrichment jobs

After IBM Software Hub is restored, running metadata enrichment jobs might not complete successfully. Such jobs must be manually restarted.

5.7 Rerunning IBM Knowledge Catalog lineage data import jobs

If a lineage data import job is running at the same time that an online backup is taken, the job is in a Complete state when the backup is restored. However, users cannot see lineage data in the catalog. Rerun the lineage import job.

5.8 Restarting IBM Knowledge Catalog lineage pods

5.9 Retraining existing watsonx Assistant skills and creating secrets to connect to Multicloud Object Gateway

After restoring the watsonx Assistant backup, it is necessary to retrain the existing skills. This involves modifying a skill, to trigger training. The training process for a skill typically requires less than 10 minutes to complete. For more information, see the Retraining your backend model section in the IBM Cloud documentation.

5.10 Creating secrets to connect Watson Discovery to Multicloud Object Gateway

5.11 Creating secrets to connect watsonx Orchestrate to Multicloud Object Gateway

5.12 Creating secrets to connect Watson Speech services to Multicloud Object Gateway

Some Watson Speech services pods might be in an Error state because they cannot connect to Multicloud Object Gateway.

oc get po -l 'app.kubernetes.io/component in (stt-models, tts-voices)' -n ${PROJECT_CPD_INST_OPERANDS} | grep ${CUSTOM_RESOURCE_SPEECH}

5.13 Installing the privileged monitoring service

If the privileged monitoring service was installed in the source cluster, install the service in the target cluster. For details, see Installing privileged monitors.

5.14 Restoring services that do not support online backup and restore