Updating Watson Machine Learning assets following an upgrade or rollback

These items provide information for working with Watson Machine Learning assets after either updating from an earlier release of Cloud Pak for Data, or rolling back to a previous version.

Choose the category that fits your use case.

• Working with assets following an upgrade
• Known issues following an upgrade
• Working with assets following a rollback
• Known issues following a rollback

Working with assets following an upgrade

• The upgrade is a disruptive process. During the upgrade process Watson Machine Learning assets and deployments will not be available for use.
• The scoring url of deployments will change but the deployment-id will remain the same.
• Existing deployments of discontinued frameworks you created in Watson Machine Learning version that is being upgraded will not be available after the upgrade. However, deployment of deprecated frameworks will be supported. Refer to Supported frameworks for a list of deprecated and discontinued frameworks as well as for information on how to migrate to a supported framework. Note: An exception to this policy is that after upgrading to 4.0.6, existing PMML deployments created with software_specification spark-mllib_2.4 will be available for predictions even though support for spark-mllib_2.4 is discontinued. To create new PMML deployments, use spark-mllib_3.0.
• Platform jobs associated with deployments which are not available after upgrade will be retained as-is. You can choose to keep them if you might roll back to a previous version Watson Machine Learning or delete them if they are no longer required.
• Because of a change to the default software specification used to train AutoAI time series experiments, you must retrain the experiment after the upgrade and then create a new deployment of the resulting model.
• After upgrading to 4.0.2 or higher from any earlier version of Cloud Pak for Data, deployments of AutoAI models trained with joined data will fail. Retrain the experiment after the upgrade and then create a new deployment of the resulting model.
• All of the Watson Machine Learning assets, including models, data assets, and apps, that you created in the earlier release will be available in the UI after the upgrade.
• The Deployments dashboard will not display run metadata in the visualizations. For example, the chart for finished runs will not show run type, such as data refinery, in the visualization. Labels will display correctly for runs created following the upgrade.
• Custom images created prior to 4.0.6 release are not compatible with the 4.0.6 or later releases. You must rebuild custom images based on the new base images published in the current release. For details and steps to resolve, see the notes in Working with custom images.
• The Watson Machine Learning API will be available at new endpoints as documented in the API documentation.

• The V4 Watson Machine Learning API and Python client library are now generally available. You might need to update the code for your assets. For example, if you encounter this error when running an existing notebook:

   WMLClientError: 'project' (MetaNames.PROJECT_UID) and 'space' (MetaNames.SPACE_UID) meta names are deprecated and considered as invalid. Instead use client.set.default_space(<SPACE_GUID>) to set the space or client.set.default_project(<PROJECT_GUID>).
  

  then remove client.repository.ModelMetaNames.SPACE_UID:space_id from metadata and use client.set.default_space('<SPACE_GUID>') or client.set.default_project('<PROJECT_GUID>').

• It is a best practice that you disable scheduled batch deployment runs before upgrading and then resume the schedule after the upgrade. If a scheduled run is in progress when the upgrade is done, the job will not be upgraded and will remain in a running state. You will have to recreate the job.

Known issues following an upgrade

Note these issues for working with deployed assets following an upgrade.

Delay for score request

Expect a brief delay of 1 to 60 seconds for the first score request after an upgrade, depending on the model framework. For some frameworks, such as SPSS modeler, the first score request for a deployed model after hibernation might result in a 504 error. If this happens, submit the request again; subsequent requests should succeed.

Error creating a deployment following upgrade

A conflict accessing the Watson machine Learning pods can occur following a refresh of Cloud Pak for Data. If you create a new deployment and you get this error:

encounters error<_Rendezvous of RPC that terminated with: status = StatusCode.UNAVAILABLE

Ask your system administrator to restart the Runtime Manager pod. This will trigger an update of the runtime pods which should resolve the issue.

Problem with RShiny pods following upgrade

If Certificate Manager is enabled for Cloud Pak for Data 4.0, RShiny pods might fail to start following an upgrade. To resolve this issue, update the internalpubkey secret on the pods. For example:

   - name: internalpubkey
        secret:
          defaultMode: 420
          secretName: internal-tls
          items:
          - key: ca.crt
            path: certificate.pem

Working with assets following a rollback

Attention: Rollback of Watson Machine Learning is dependent on common core services (CCS) and Watson Studio services. Rollback of those components to 4.0.0 is not supported, so rolling back Watson Machine Learningfrom 4.0.x to 4.0.0 is not recommended and is not fully supported at this time.

If you do roll back to a previous release, note these issues for working with deployed assets following a rollback.

• The rollback is a disruptive rollback. During the rollback process Watson Machine Learning assets and deployments will not be available for use.
• The scoring url of deployments may change but the deployment ID will remain the same.
• Existing deployments with supported or deprecated frameworks will continue to be available after rollback. Deployments with unsupported frameworks will not be available. Refer to Supported frameworks for a list of deprecated and discontinued frameworks.
• You can use existing platform jobs associated with deployments that are available after a rollback.
• All of the assets you created in the higher release will be available in the UI after the rollback, but the deployments dashboard will not display run metadata in the visualizations. For example, the chart for finished runs will not show run type, such as data refinery, in the visualization. Labels will display correctly for runs created following the upgrade.
• The Watson Machine Learning API will be available at new endpoints as documented in the API documentation.
• Expect a brief delay of 1 to 60 seconds for the first score request after rollback, depending on the model framework. For some frameworks, such as SPSS modeler, the first score request for a deployed model after hibernation might result in a 504 error. If this happens, submit the request again; subsequent requests should succeed.
• It is a best practice that you disable scheduled batch deployment runs before rollback and then resume the schedule after the rollback. If a scheduled run is in progress when the rollback is done, the job will not be updated and will remain in a running state. You will have to recreate the job.

Known issues following a rollback

Rollback of Watson Machine Learning is dependent on common core services (CCS) and Watson Studio services. Rollback of those components to 4.0.0 is not supported, so rolling back Watson Machine Learningfrom 4.0.x to 4.0.0 is not recommended and is not fully supported. If you attempt a rollback, Watson Machine Learning deployments will be in an inconsistent state. Note these issue for working with deployed assets following a rollback to a previous version from an upgrade.

Rollback of WML from 4.0.x to 4.0.0

Consider the following before rolling back:

1. Watson Machine Learning online deployments will be in inconsistent state if either of the following is true:
   • You submitted a score request on an existing Watson Machine Learning online deployments after the upgrade to 4.0.x.
   • You created new Watson Machine Learning online deployments of any of the supported frameworks in 4.0.x.
2. If the framework version for a deployment is supported in both 4.0.0 and 4.0.x, and you do not attempt scoring requests in 4.0.x, then scoring will work when you rollback to 4.0.0. If not, scoring will fail with an error.
3. Existing deployments with model frameworks that are not supported in 4.0.x are cleaned during upgrade to 4.0.x. Those deployments will not be available after a rollback to 4.0.0. Creation of new deployment for such models is not allowed as dependent components in CCS and Watson Studio are not rolled back to 4.0.0. 4.R-Shiny deployments after rollback from 4.0.x to 4.0.0 will not work. You must redeploy R-Shiny applications.

Rollback of Watson Machine Learning to different version other than from where it is upgraded

You can potentially roll back Watson Machine Learning to a different version from the one you upgraded from. For example, if you upgraded from 4.0.x to 4.0.y and roll back from 4.0.y to 4.0.z. However, this is not recommended.

You can roll back Watson Machine Learning to 4.0.z but Watson Machine Learning roll back would not know the state to restore the deployments in 4.0.z if you upgraded from a different version (for example, 4.0.x). If the framework version for a deployment is supported in 4.0.z, then scoring will work. If not, scoring will fail with an unsupported framework version error.

Parent topic: Managing assets