IBM Cloud Pak foundational services backup and restore

You can schedule backup and restore of foundational services by using the Red Hat OADP (OpenShift API for Data Protection) operator. Make sure that you use the stable-1.1 channel of the OADP operator.

Note: If you are running the scripts in an air-gapped environment, you must first download the scripts from the following locations to the air-gapped environment:

Prerequisites

Set up any Amazon S3-compatible storage. For example, you can create a bucket in IBM Cloud Object Storage. For more information, see IBM Cloud Object Storage Opens in a new tab .

When you add a service credential to the bucket, include the hash-based message authentication code (HMAC). For more information, see Service credentials Opens in a new tab . From the Cloud Object Storage navigation menu, gather the following information:

access key id, which can be found on the Service credentials page that is associated with the bucket.
secret access key, which can be found on the Service credentials page that is associated with the bucket.
bucket name, which can be found on the Buckets page.
bucket region, which can be found on the Buckets page.
root directory name, which is the path in the bucket where you want to store the backups.
s3 URL, which is the endpoint URL of the bucket. Note: The endpoint URL must start with http:// or https://.

Backing up foundational services

Complete the following steps to back up the installed foundational services.

Create the backup resources

You need the following resources for completing the backup procedures.

Log in to your OpenShift cluster command-line interface (CLI) by using the oc login command.
Create a namespace for Velero objects. The following example creates the velero namespace. For more information about Velero, see Velero documentation .
```
 oc project velero
```
Install the Red Hat OADP operator in the velero namespace. For more information, see About installing OADP .
Create a secret named cloud-credentials with the access key id and secret access key credentials.
1. Open any editor and place the following credentials in a file named credentials-velero.
```
 vi credentials-velero
```
2. Insert the following content in the file.
```
 [default]
 aws_access_key_id=<access_key_id>
 aws_secret_access_key=<secret_access_key>
```
3. Create the secret.
```
oc create secret generic cloud-credentials -n velero --from-file cloud=credentials-velero
```
From your OpenShift cluster console OperatorHub page, install the OADP operator from the stable-1.1 channel, which provides the Velero 1.9 API. The API is needed for foundational services backup and restore. For more information, see Red Hat® OpenShift® Container Platform documentation .
Create a DataProtectionApplication object.

Note: The provider is aws even if you are not using AWS Object Storage.

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: <resource_name>
  namespace: velero
spec:
  backupLocations:
    - velero:
        config:
          profile: default
          region: <bucket_region>
          s3ForcePathStyle: 'true'
          s3Url: <s3_URL>
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: <bucket_name>
          prefix: <root_directory_name>
        provider: aws
  configuration:
    restic:
      enable: true
    velero:
      defaultPlugins:
        - openshift
        - aws
      podConfig:
        resourceAllocations:
          limits:
            cpu: '1'
            memory: 1Gi
          requests:
            cpu: 500m
            memory: 512Mi

Add labels to resources

You need to label the currently installed resources to identify them during restoration.

Before you begin, set the namespace where you installed foundational services as the default namespace.

oc project <namespace-where-foundational services-are-installed>

Add labels to the Licensing service configmaps:

Get the script to add the labels.

 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/backup/licensing/label-licensing-configmaps.sh

Add executable permission.

 chmod +x label-licensing-configmaps.sh

Run the script.
```
 ./label-licensing-configmaps.sh
```

Add labels to the Cert Manager resources:

Get the script to add the labels.

 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/backup/cert-manager/label-cert-manager.sh

Add executable permission.
```
 chmod +x label-cert-manager.sh
```
Run the script.
```
 ./label-cert-manager.sh
```

Add a label to the catalog source:

oc label catalogsource ibm-operator-catalog foundationservices.cloudpak.ibm.com=catalog -n openshift-marketplace --overwrite=true

Add a label to the operator group:

Get the name of the operator group.

 oc get operatorgroup -n <namespace-where-foundational services-are-installed>

Add a label.

 oc label operatorgroup <operatorgroup-name> foundationservices.cloudpak.ibm.com=operatorgroup --overwrite=true

Add labels to the IBM Common Service Operator subscription:

oc label subscription ibm-common-service-operator foundationservices.cloudpak.ibm.com=subscription --overwrite=true

Add a label to the common-service custom resource (CR):

  oc label commonservices common-service foundationservices.cloudpak.ibm.com=commonservice --overwrite=true

Add a label to the entitlement secret, if you have one in your cluster:

oc label secret ibm-entitlement-key foundationservices.cloudpak.ibm.com=entitlementkey --overwrite=true

Add a label to the global pull secret, if you have one in your cluster:

oc label secret pull-secret -n openshift-config foundationservices.cloudpak.ibm.com=pull-secret --overwrite=true

Add a label to the namespace where you installed foundational services.

oc label namespace <namespace-where-foundational services-are-installed> foundationservices.cloudpak.ibm.com=namespace --overwrite=true

Add a label to the OperandRequest:

oc label operandrequests common-service foundationservices.cloudpak.ibm.com=operand --overwrite=true

If you have a customized OperandConfig, add a label to the OperandConfig:

oc label operandconfig common-service foundationservices.cloudpak.ibm.com=operand --overwrite=true

If Zen is installed, add foundationservices.cloudpak.ibm.com=zen to the ZenService and the Zen OperandRequest.
- To find the Zen service and namespace, run the following command:
```
  oc get zenservice -A
```
- To find the zen OperandRequest, run the following command. The | grep zen filter is used in the following command on the assumption that your zen operandRequest name has the word zen in it.
```
  oc get operandrequest -A | grep zen
```

Back up MongoDB

Set up the MongoDB backup deployment. The deployment triggers and holds the MongoDB database backup. In the event of a restore, the deployment triggers the restore of the MongoDB database as well.

Get the mongodb-backup-pvc.yaml and mongodb-backup-deployment.yaml files. Place them in the same directory.

 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/schedule/mongodb-backup-pvc.yaml

 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/schedule/mongodb-backup-deployment.yaml

Update the backup files.

By default, the mongodb-backup-pvc.yaml file is set up to run in the ibm-common-services namespace and to use the ibmc-block-retain-gold storage class. Replace the following parameters, as needed:
- Replace ibm-common-services with the namespace where the MongoDB deployment that foundational services uses is running.
- Replace the StorageClassName with either the storage class that MongoDB deployment uses, or with any storage class that has Retain ReclaimPolicy.
Add the PVC to the cluster.
```
 oc apply -f mongodb-backup-pvc.yaml
```

Add the deployment to the cluster.

 oc apply -f mongodb-backup-deployment.yaml

Back up Zen MetastoreDB

The instructions to back up the Zen metastoreDB are based on the assumption that the Platform UI (Zen) is installed in the same namespace as foundational services. If that is not the case, then use the namespace where Platform UI is installed in the following commands.

Get the zen_backup_pvc.yaml and zen-backup-deployment.yaml files.

 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/schedule/zen-backup-pvc.yaml

 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/schedule/zen-backup-deployment.yaml

Update the backup files.

By default, the zen-backup-pvc.yaml file is set up to run in the ibm-common-services namespace and to use the ibmc-block-retain-gold storage class. You must replace the following parameters, as needed:
- Replace ibm-common-services with the namespace where the Zen service is deployed.
- Replace the StorageClassName with either the storage class that zen-metastoredb deployment uses, or with any storage class that has Retain ReclaimPolicy.
Create the PVC.
```
 oc apply -f zen-backup-pvc.yaml
```
Add the Zen backup deployment to the cluster.
```
 oc apply -f zen-backup-deployment.yaml
```

Create a backup resource

Create a backup resource for the velero namespace.

Get the schedule-common-services.yaml file.

 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/schedule/schedule-common-services.yaml

Update the schedule-common-services.yaml file based on your backup requirements. For more information, see Velero Schedule API Type . By default, the backup runs once a day and is deleted 48 hours later.

The following configurations in the schedule-common-services.yaml file are important:
- schedule:, which is a CRON expression. CRON uses the server time, which is usually the Coordinated Universal Time unless configured to be something else.
- ttl, which is the time to live for the backup.
- storageLocation, which is the same storage location that you used when you set up OADP.
- velero, which is the namespace where you installed OADP.

Create the resource.

 oc apply -f schedule-common-services.yaml

Verify whether the backup schedule ran.
```
 velero schedule get
```

After the first scheduled time passes, you can verify whether the backup ran. Look for a schedule name and timestamp.

velero backup get

Restoring foundational services

Complete the following steps to restore foundational services.

Before you restore foundational services, set up Velero on the new cluster. Follow the instructions in the Create the backup resources section.

Restore the foundational services namespace by using the restore-namespace.yaml file.
1. Get the name of the Velero backup that you plan to use for restoring.
```
 velero backup get
```
  Replace __BACKUP_NAME__ in the following commands with the Velero backup name.
2. Get the namespace restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-namespace.yaml
```
3. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-namespace.yaml
```
4. Restore the namespace.
```
 oc apply -f restore-namespace.yaml
```
  You can check the progress of the restore by using the following command. Proceed with the next step after the status shows as Completed.
```
 velero restore get
```
5. Verify whether the namespace is restored. Your namespace must be listed in the command output.
```
 oc get namespace
```
  Proceed with the next step after the namespace is restored.
6. Change the default project to the restored namespace.
```
 oc project <namespace-where-foundational services-are-installed>
```
Restore the entitlement key.
1. Get the entitlement key restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-entitlementkey.yaml
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-entitlementkey.yaml
```
3. Restore the entitlement key.
```
 oc apply -f restore-entitlementkey.yaml
```
4. Check restore progress. Proceed with the next step after restore is complete.
```
 velero restore get
```
5. Verify whether the entitlement key is restored.
```
 oc get secret
```

Restore the pull secret.

Save the current pull secret.

 oc get secret pull-secret -n openshift-config -o yaml > original-pull-secret.yaml

Delete the current pull secret from the openshift-config namespace.
```
 oc delete secret pull-secret -n openshift-config
```

Get the pull secret restore resource.

 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-pull-secret.yaml

Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-pull-secret.yaml
```
Restore the pull secret.
```
 oc apply -f restore-pull-secret.yaml
```
Check restore progress. Proceed with the next step after restore is complete.
```
 velero restore get
```

Verify whether the pull secret is restored.

 oc get secret -n openshift-config | grep pull

Restore the catalog.
1. Get the catalog restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-catalog.yaml
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-catalog.yaml
```
3. Restore the catalog.
```
 oc apply -f restore-catalog.yaml
```
4. Check restore progress. Proceed with the next step after restore is complete.
```
 velero restore get
```
5. Verify whether the catalogsource is restored.
```
 oc get catalogsource -n openshift-marketplace | grep ibm
```
6. Verify whether the ibm-operator-catalog pod is running.
```
 oc get pod -n openshift-marketplace | grep ibm-operator-catalog | grep Running
```
  If the pod is running, proceed with the next step.
Restore the operatorgroup.
1. Get the operatorgroup restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-operatorgroup.yaml
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-operatorgroup.yaml
```
3. Restore the operatorgroup.
```
 oc apply -f restore-operatorgroup.yaml
```
4. Check restore progress. Proceed with the next step after restore is complete.
```
 velero restore get
```
5. Verify whether the operatorgroup is restored.
```
 oc get operatorgroup
```
Restore the subscriptions.
1. Get the subscriptions restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-subscriptions.yaml
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-subscriptions.yaml
```
3. Restore the subscriptions.
```
 oc apply -f restore-subscriptions.yaml
```
4. Check restore progress. Proceed with the next step after restore is complete.
```
 velero restore get
```
5. Watch the foundational services namespace for the operand-deployment-lifecycle-manager to be running.
```
 watch -n 10 oc get pod | grep operand-deployment-lifecycle-manager | grep Running
```
Restore the common-service CR.
1. Delete the current common-service CR.
```
 oc delete commonservices common-service
```
2. Get the common-service CR restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-commonservice.yaml
```
3. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-commonservice.yaml
```
4. Restore the CR.
```
 oc apply -f restore-commonservice.yaml
```
5. Check restore progress. Proceed with the next step after restore is complete.
```
 velero restore get
```
6. Verify whether the commonservice is restored.
```
 oc get commonservice
```
  If the foundational services are not restored, delete the restore resource and apply it again.
  1. Delete the resource.
```
 oc delete -f restore-commonservice.yaml
```
  2. Restore the CR.
```
 oc apply -f restore-commonservice.yaml
```
    Wait for 30 seconds and check again for the CommonService resource.
Restore Licensing service configmap.
1. Get the Licensing service configmap restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-licensing.yaml
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-licensing.yaml
```
3. Restore the configmap.
```
 oc apply -f restore-licensing.yaml
```
4. Check restore progress. Proceed with the next step after restore is complete.
```
 velero restore get
```
5. Verify whether the configmap is restored.
```
 oc get configmap | grep licensing
```
Restore cert manager resource.
1. Get the cert manager restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-cert-manager.yaml
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-cert-manager.yaml
```
3. Restore the cert manager resource.
```
 oc apply -f restore-cert-manager.yaml
```
4. Check restore progress. Proceed with the next step after restore is complete.
```
 velero restore get
```
5. Verify whether the certificates are restored.
```
 oc get certificates
```
Restore the OperandRequest and OperandConfig.
1. Get the operands restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-operands.yaml
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-operands.yaml
```
3. Restore the operands.
```
 oc apply -f restore-operands.yaml
```
4. Check restore progress. Proceed with the next step after restore is complete.
```
 velero restore get
```
5. Verify whether the operands are restored.
```
 oc get operandrequest
```
```
 oc get operandconfig
```
Restore Platform UI (Zen) resources.

Note: If Zen is installed in a namespace other than the namespace where foundational services are installed, then create that namespace first.
```
oc new-project <namespace-where-Zen-is-installed>
```
1. Get the Zen restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-zen.yaml
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-zen.yaml
```
3. Restore Zen resources.
```
 oc apply -f restore-zen.yaml
```
  Wait for the Zen Service to be available. When the route with cp4d in its name is ready, then the Zen is available.
```
 oc get route -n <namespace-where-Zen-is-installed> | grep cpd
```
4. Check restore progress. Proceed with the next step after restore is complete.
```
 velero restore get
```
5. Verify whether the zenservice is restored.
```
 oc get zenservice
```
Restore data to IAM and MongoDB data.
1. Get the data restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-mongo-data.yaml
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-mongo-data.yaml
```
3. Restore data.
```
 oc apply -f restore-mongo-data.yaml
```
4. Verify whether the restore is complete. It might take a few minutes to restore the data to the cluster. Proceed with the next step after restore is complete.
```
 velero restore get
```
5. Verify whether the mongodb-backup pod is running.
```
 oc get pod | grep mongodb-backup
```
Restore Zen data.
1. Get the data restore resource.
```
 wget https://raw.githubusercontent.com/IBM/ibm-common-service-operator/scripts/velero/restore/restore-zen-data.yaml
```
2. Substitute the __BACKUP_NAME__ with the name of the backup resource that you created in a previous step.
```
 vi restore-zen-data.yaml
```
3. Restore data.
```
 oc apply -f restore-zen-data.yaml
```
4. Verify whether the restore is complete. It might take a few minutes to restore the data to the cluster. Proceed with the next step after restore is complete.
```
 velero restore get
```
5. Verify whether the zen-backup pod is running. The pod runs in the same namespace as the zen service. You can use -n <zen-service-namespace>, if applicable, in the following command:
```
 oc get pod | grep zen-backup
```

All restoration tasks are completed.

Verify whether foundational services are properly restored.

Verify the pods:

oc get pods

All pods must be running.

Verify the subscriptions:

oc get subscriptions

Subscriptions of all installed services must be listed.

Verify that the Identity and Access section of the cp-console shows the users and teams that your organization added in the original cluster.