Backing up and restoring the IBM Watson Machine Learning Accelerator service

Edit online

Use this information to backup or restore the IBM Watson® Machine Learning Accelerator service.

Backing up IBM Watson Machine Learning Accelerator

Before you back up the IBM Watson Machine Learning Accelerator service using the standard backup process, you must stop all running workloads and delete your conda content.

Steps:
  1. Stop all running workloads.
    1. As an IBM Watson Machine Learning Accelerator project administrator, from the IBM Watson Machine Learning Accelerator console stop all running jobs.
      1. Log in to the IBM Watson Machine Learning Accelerator console as a project administrator.
      2. Navigate to Monitoring > Applications.
      3. For each running applications, select the menu icon and click Stop.
    2. Stop all running deployed models. Use the WML Accelerator console or the command line interface to stop each running model, see Stop an inference service.
  2. If you are using Portworx or OCS, delete the conda content from the persistent volume (PV):
    1. Log in to your OpenShift cluster as a project administrator.
      oc login OpenShift_URL:port
    2. Switch to the WML Accelerator namespace.
      oc project wmla-namespace
    3. Get the conda pod name:
      oc get po |grep wmla-conda
    4. Delete conda PV data from wmla-conda pod:
      oc exec -it wmla-conda-pod-name bash
bash-4.4$ rm -rfv /opt/conda/*
    5. Delete the conda_synced file from the wmla-conda pod: 
      oc exec -it wmla-conda-pod-name bash
bash-4.4$ rm -rf /var/shareDir/dli/work/conda_synced
  3. Back up the IBM Watson Machine Learning Accelerator service using the standard backup process. See: https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=project-backing-up

Restoring IBM Watson Machine Learning Accelerator

After you restore the IBM Watson Machine Learning Accelerator service using the standard restore process, you must run an additional script to restore owner references to all IBM Watson Machine Learning Accelerator resources.

Steps:
  1. Complete the standard restore process. See: https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=project-restoring
  2. Log in to your OpenShift cluster as a project administrator.
    oc login OpenShift_URL:port
  3. Update the IBM NamespaceScope Operator in the IBM Cloud Pak for Data platform operator project to watch the project where Watson Machine Learning Accelerator is installed.
    oc edit namespacescope -n ibm-common-services common-service
  4. Switch to the IBM Watson Machine Learning Accelerator namespace.
    oc project wmla-namespace
  5. Return owner references to IBM Watson Machine Learning Accelerator resources, run the following script:
    #!/bin/bash
wmla_name=`oc get wmla -o name|awk -F/ '{print $NF}'`
wmla_uid=`oc get wmla $wmla_name -o jsonpath='{.metadata.uid}'`

for r in \
deployment.apps/wmla-auth-rest \
deployment.apps/wmla-conda \
deployment.apps/wmla-dlpd \
deployment.apps/wmla-edi-imd \
deployment.apps/wmla-edi-lbd \
deployment.apps/wmla-grafana \
deployment.apps/wmla-gui \
deployment.apps/wmla-infoservice \
deployment.apps/wmla-ingress \
deployment.apps/wmla-jupyter-gateway \
deployment.apps/wmla-jupyter-hub \
deployment.apps/wmla-jupyter-proxy \
deployment.apps/wmla-logstash \
deployment.apps/wmla-msd \
deployment.apps/wmla-mss \
deployment.apps/wmla-prometheus \
statefulset.apps/wmla-etcd \
statefulset.apps/wmla-mongodb \
role.rbac.authorization.k8s.io/wmla-edi \
role.rbac.authorization.k8s.io/wmla-msd-mss \
role.rbac.authorization.k8s.io/wmla-role \
rolebinding.rbac.authorization.k8s.io/wmla-edi \
rolebinding.rbac.authorization.k8s.io/wmla-msd-mss \
rolebinding.rbac.authorization.k8s.io/wmla-rb \
certificate.cert-manager.io/wmla-ca-crt \
certificate.cert-manager.io/wmla-internal-keys \
certificate.cert-manager.io/wmla-nginx-keys \
certificate.certmanager.k8s.io/wmla-ca-crt \
certificate.certmanager.k8s.io/wmla-internal-keys \
certificate.certmanager.k8s.io/wmla-nginx-keys \
issuer.cert-manager.io/wmla-ca \
issuer.cert-manager.io/wmla-root-issuer \
issuer.certmanager.k8s.io/wmla-ca \
issuer.certmanager.k8s.io/wmla-root-issuer \
ingress.extensions/wmla-jupyter-ingress \
networkpolicy.networking.k8s.io/wmla-ingress-network-policy \
networkpolicy.networking.k8s.io/wmla-logstash-network-policy \
networkpolicy.networking.k8s.io/wmla-namespace-network-policy \
poddisruptionbudget.policy/wmla-jupyter-hub-pdb \
poddisruptionbudget.policy/wmla-jupyter-proxy-pdb \
role.rbac.authorization.k8s.io/wmla-edi \
role.rbac.authorization.k8s.io/wmla-msd-mss \
role.rbac.authorization.k8s.io/wmla-role \
rolebinding.rbac.authorization.k8s.io/wmla-edi \
rolebinding.rbac.authorization.k8s.io/wmla-msd-mss \
rolebinding.rbac.authorization.k8s.io/wmla-rb \
route.route.openshift.io/wmla-console \
route.route.openshift.io/wmla-grafana \
route.route.openshift.io/wmla-inference \
route.route.openshift.io/wmla-jupyter-notebook \
configmap/cpd-wmla-br-cm \
configmap/cpd-wmla-qu-cm \
configmap/product-configmap \
configmap/wmla-dlpd-bootstrap \
configmap/wmla-edi \
configmap/wmla-edi-imd-nginx \
configmap/wmla-edi-isd \
configmap/wmla-edi-isd-ingress \
configmap/wmla-grafana-configmap \
configmap/wmla-grafana-ini \
configmap/wmla-grafana-providers \
configmap/wmla-infoservice \
configmap/wmla-jupyter-hub-config \
configmap/wmla-logstash-conf \
configmap/wmla-mongodb-shells \
configmap/wmla-msd \
configmap/wmla-mss \
configmap/wmla-nginx-conf \
configmap/wmla-nginx-grafana-sidecar-conf \
configmap/wmla-nginx-sidecar-conf \
configmap/wmla-prometheus \
configmap/wmla-version-info \
persistentvolumeclaim/wmla-conda \
persistentvolumeclaim/wmla-cws-share \
persistentvolumeclaim/wmla-edi \
persistentvolumeclaim/wmla-infoservice \
persistentvolumeclaim/wmla-logging \
persistentvolumeclaim/wmla-mygpfs \
secret/internal-tls \
secret/wmla-dlpd-conf \
secret/wmla-eg-secret \
secret/wmla-jupyter-hub-secret \
secret/wmla-mongodb-secret \
secret/wmla-prometheus-htpasswd \
service/wmla-auth-rest \
service/wmla-dlpd \
service/wmla-edi \
service/wmla-etcd \
service/wmla-grafana \
service/wmla-gui \
service/wmla-inference \
service/wmla-infoservice \
service/wmla-ingress \
service/wmla-jupyter-enterprise-gateway \
service/wmla-jupyter-hub \
service/wmla-jupyter-proxy-api \
service/wmla-jupyter-proxy-public \
service/wmla-logstash-service \
service/wmla-mongodb \
service/wmla-msd \
service/wmla-mss \
service/wmla-prometheus \
serviceaccount/wmla-msd-mss \
serviceaccount/wmla-norbac \
serviceaccount/wmla-sa;
do
	oc get $r >& /dev/null
	if [ $? == "0" ]; then
		echo "Patch ownerReferences for $r"
		oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}"
	fi
done

#update ownerReferences for wmla resource plans
wmla_rps=`oc get rp -o name|grep -v 'wml-accelerator'`
for r in $wmla_rps;
do
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}"
done

#update ownerReferences for deploy/isd and isd/service
isds=`oc get deploy -o name|grep wmla-edi-isd`
imd_uid=`oc get deploy wmla-edi-imd -o jsonpath='{.metadata.uid}'`
for r in $isds;
do
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"wmla-edi-imd\",\"uid\":\"$imd_uid\"}]}}"
done

isd_servicess=`oc get services -o name|grep wmla-edi-isd`
for r in $isd_servicess;
do
	isd_name=`echo $r|awk -F/ '{print $NF}'`
	isd_uid=`oc get deploy $isd_name -o jsonpath='{.metadata.uid}'`
	echo "Patch ownerReferences for $r"
	oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"$isd_name\",\"uid\":\"$isd_uid\"}]}}"
done