Backing up and restoring the IBM Watson Machine Learning Accelerator service
Use this information to backup or restore the IBM Watson® Machine Learning Accelerator service.
Backing up IBM Watson Machine Learning Accelerator
Before you back up the IBM Watson Machine Learning Accelerator service using the standard backup process, you must stop all running workloads and delete your conda content.
Steps:
- Stop all running workloads.
- As an IBM Watson Machine Learning Accelerator project
administrator, from the IBM Watson Machine Learning Accelerator console stop
all running jobs.
- Log in to the IBM Watson Machine Learning Accelerator console as a project administrator.
- Navigate to .
- For each running applications, select the menu icon and click Stop.
- Stop all running deployed models. Use the WML Accelerator console or the command line interface to stop each running model, see Stop an inference service.
- As an IBM Watson Machine Learning Accelerator project
administrator, from the IBM Watson Machine Learning Accelerator console stop
all running jobs.
- If you are using Portworx or OCS, delete the conda content from the persistent volume (PV):
- Log in to your OpenShift cluster as a project
administrator.
oc login OpenShift_URL:port
- Switch to the WML Accelerator namespace.
oc project wmla-namespace
- Get the conda pod
name:
oc get po |grep wmla-conda
- Delete conda PV data from wmla-conda
pod:
oc exec -it wmla-conda-pod-name bash bash-4.4$ rm -rfv /opt/conda/*
- Delete the conda_synced file from the wmla-conda pod:
oc exec -it wmla-conda-pod-name bash bash-4.4$ rm -rf /var/shareDir/dli/work/conda_synced
- Log in to your OpenShift cluster as a project
administrator.
- Back up the IBM Watson Machine Learning Accelerator service using the standard backup process. See: https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=project-backing-up
Restoring IBM Watson Machine Learning Accelerator
After you restore the IBM Watson Machine Learning Accelerator service using the standard restore process, you must run an additional script to restore owner references to all IBM Watson Machine Learning Accelerator resources.
Steps:
- Complete the standard restore process. See: https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=project-restoring
- Log in to your OpenShift cluster as a project
administrator.
oc login OpenShift_URL:port
- Update the
IBM NamespaceScope Operator
in the IBM Cloud Pak for Data platform operator project to watch the project where Watson Machine Learning Accelerator is installed.oc edit namespacescope -n ibm-common-services common-service
- Switch to the IBM Watson Machine Learning Accelerator
namespace.
oc project wmla-namespace
- Return owner references to IBM Watson Machine Learning Accelerator
resources, run the following script:
#!/bin/bash wmla_name=`oc get wmla -o name|awk -F/ '{print $NF}'` wmla_uid=`oc get wmla $wmla_name -o jsonpath='{.metadata.uid}'` for r in \ deployment.apps/wmla-auth-rest \ deployment.apps/wmla-conda \ deployment.apps/wmla-dlpd \ deployment.apps/wmla-edi-imd \ deployment.apps/wmla-edi-lbd \ deployment.apps/wmla-grafana \ deployment.apps/wmla-gui \ deployment.apps/wmla-infoservice \ deployment.apps/wmla-ingress \ deployment.apps/wmla-jupyter-gateway \ deployment.apps/wmla-jupyter-hub \ deployment.apps/wmla-jupyter-proxy \ deployment.apps/wmla-logstash \ deployment.apps/wmla-msd \ deployment.apps/wmla-mss \ deployment.apps/wmla-prometheus \ statefulset.apps/wmla-etcd \ statefulset.apps/wmla-mongodb \ role.rbac.authorization.k8s.io/wmla-edi \ role.rbac.authorization.k8s.io/wmla-msd-mss \ role.rbac.authorization.k8s.io/wmla-role \ rolebinding.rbac.authorization.k8s.io/wmla-edi \ rolebinding.rbac.authorization.k8s.io/wmla-msd-mss \ rolebinding.rbac.authorization.k8s.io/wmla-rb \ certificate.cert-manager.io/wmla-ca-crt \ certificate.cert-manager.io/wmla-internal-keys \ certificate.cert-manager.io/wmla-nginx-keys \ certificate.certmanager.k8s.io/wmla-ca-crt \ certificate.certmanager.k8s.io/wmla-internal-keys \ certificate.certmanager.k8s.io/wmla-nginx-keys \ issuer.cert-manager.io/wmla-ca \ issuer.cert-manager.io/wmla-root-issuer \ issuer.certmanager.k8s.io/wmla-ca \ issuer.certmanager.k8s.io/wmla-root-issuer \ ingress.extensions/wmla-jupyter-ingress \ networkpolicy.networking.k8s.io/wmla-ingress-network-policy \ networkpolicy.networking.k8s.io/wmla-logstash-network-policy \ networkpolicy.networking.k8s.io/wmla-namespace-network-policy \ poddisruptionbudget.policy/wmla-jupyter-hub-pdb \ poddisruptionbudget.policy/wmla-jupyter-proxy-pdb \ role.rbac.authorization.k8s.io/wmla-edi \ role.rbac.authorization.k8s.io/wmla-msd-mss \ role.rbac.authorization.k8s.io/wmla-role \ rolebinding.rbac.authorization.k8s.io/wmla-edi \ rolebinding.rbac.authorization.k8s.io/wmla-msd-mss \ rolebinding.rbac.authorization.k8s.io/wmla-rb \ route.route.openshift.io/wmla-console \ route.route.openshift.io/wmla-grafana \ route.route.openshift.io/wmla-inference \ route.route.openshift.io/wmla-jupyter-notebook \ configmap/cpd-wmla-br-cm \ configmap/cpd-wmla-qu-cm \ configmap/product-configmap \ configmap/wmla-dlpd-bootstrap \ configmap/wmla-edi \ configmap/wmla-edi-imd-nginx \ configmap/wmla-edi-isd \ configmap/wmla-edi-isd-ingress \ configmap/wmla-grafana-configmap \ configmap/wmla-grafana-ini \ configmap/wmla-grafana-providers \ configmap/wmla-infoservice \ configmap/wmla-jupyter-hub-config \ configmap/wmla-logstash-conf \ configmap/wmla-mongodb-shells \ configmap/wmla-msd \ configmap/wmla-mss \ configmap/wmla-nginx-conf \ configmap/wmla-nginx-grafana-sidecar-conf \ configmap/wmla-nginx-sidecar-conf \ configmap/wmla-prometheus \ configmap/wmla-version-info \ persistentvolumeclaim/wmla-conda \ persistentvolumeclaim/wmla-cws-share \ persistentvolumeclaim/wmla-edi \ persistentvolumeclaim/wmla-infoservice \ persistentvolumeclaim/wmla-logging \ persistentvolumeclaim/wmla-mygpfs \ secret/internal-tls \ secret/wmla-dlpd-conf \ secret/wmla-eg-secret \ secret/wmla-jupyter-hub-secret \ secret/wmla-mongodb-secret \ secret/wmla-prometheus-htpasswd \ service/wmla-auth-rest \ service/wmla-dlpd \ service/wmla-edi \ service/wmla-etcd \ service/wmla-grafana \ service/wmla-gui \ service/wmla-inference \ service/wmla-infoservice \ service/wmla-ingress \ service/wmla-jupyter-enterprise-gateway \ service/wmla-jupyter-hub \ service/wmla-jupyter-proxy-api \ service/wmla-jupyter-proxy-public \ service/wmla-logstash-service \ service/wmla-mongodb \ service/wmla-msd \ service/wmla-mss \ service/wmla-prometheus \ serviceaccount/wmla-msd-mss \ serviceaccount/wmla-norbac \ serviceaccount/wmla-sa; do oc get $r >& /dev/null if [ $? == "0" ]; then echo "Patch ownerReferences for $r" oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}" fi done #update ownerReferences for wmla resource plans wmla_rps=`oc get rp -o name|grep -v 'wml-accelerator'` for r in $wmla_rps; do echo "Patch ownerReferences for $r" oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"spectrumcomputing.ibm.com/v1\",\"kind\":\"Wmla\",\"name\":\"$wmla_name\",\"uid\":\"$wmla_uid\"}]}}" done #update ownerReferences for deploy/isd and isd/service isds=`oc get deploy -o name|grep wmla-edi-isd` imd_uid=`oc get deploy wmla-edi-imd -o jsonpath='{.metadata.uid}'` for r in $isds; do echo "Patch ownerReferences for $r" oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"wmla-edi-imd\",\"uid\":\"$imd_uid\"}]}}" done isd_servicess=`oc get services -o name|grep wmla-edi-isd` for r in $isd_servicess; do isd_name=`echo $r|awk -F/ '{print $NF}'` isd_uid=`oc get deploy $isd_name -o jsonpath='{.metadata.uid}'` echo "Patch ownerReferences for $r" oc patch $r --type merge -p "{\"metadata\":{\"ownerReferences\":[{\"apiVersion\":\"apps/v1\",\"blockOwnerDeletion\":true,\"controller\":true,\"kind\":\"Deployment\",\"name\":\"$isd_name\",\"uid\":\"$isd_uid\"}]}}" done