Creating a custom software specification in a project

This topic contains information on how to create a custom software specification in a project and then promote it to a deployment space.

If your model requires custom components such as user-defined transformers, estimators, or user-defined tensors, you can create a custom software specification derived from a base, or a predefined specification. Python functions and python scripts also support custom software specifications.

You can use custom software specification to reference any third-party libraries, user-created python packages, or both. Third-party libraries or user-created python packages must be specified as package extensions which can then be referenced in a custom software specification.

Custom software specifications are promoted with the assets that use them. Python functions and scripts, and models created with the Scikit-learn, XGBoost and Tensorflow frameworks can use the software specifications for deployments.

Note that software specifications are also included when you import a project or space that includes one.

For details on creating environments with software specifications, see Environments. For details on adding custom components, see Custom components.

Overview of creating a custom software specification

The high-level steps to create a custom software specification that uses third-party libraries and user-created python packages.

Create a custom software specification
Save a conda YAML file that contains a list of 3rd party libraries and create a package extension. Note: This step is not required if the model does not have any dependency on a third-party library.
Add a reference of the package extensions to the custom software specification you created.

Creating a custom software specification in a notebook

You can use the Watson Machine Learning APIs or python client to define a custom software specification that is derived from a base specification.

These notes are applicable when specifying software specification for Scikit-Learn, XGBoost, Tensorflow, Keras, PyTorch, or Caffe models trained in Watson Studio notebooks, or python functions developed in a notebook.

If you have created the models or python functions in a Watson Studio notebook with "Default Python 3.9" environment, then save the model or function with a reference to the software specification using the name "default_py3.9". To get the corresponding software specification id, use:
```
 softwareSpecId = wml_client.software_specifications.get_id_by_name('default_py3.9')
```
You can create custom environments in Watson Studio to install third-party libraries required to execute your python scripts. If you have trained a model or developed a python function in a Watson Studio notebook with customized environment, specify the reference to the custom software specification when you save the model or function. The name of the custom software specification will be the same as the name of the custom environment. For example, if the custom environment is called "my_cust_model1_env":
```
  softwareSpecId = wml_client.software_specifications.get_id_by_name('my_cust_model1_env')
```

To create a custom software specification

This code template illustrates how to create a custom software specification using the python client.

Authenticate and create the client.

Refer to Authentication.

Create and set the default deployment space, then list available software specifications.

 metadata = {
     wml_client.spaces.ConfigurationMetaNames.NAME:
         'examples-create-software-spec',
     wml_client.spaces.ConfigurationMetaNames.DESCRIPTION:
         'For my models'
 }
 space_details = wml_client.spaces.store(meta_props=metadata)
 space_uid = wml_client.spaces.get_id(space_details)

 # set the default space
 wml_client.set.default_space(space_uid)

 # see available meta names for software specs
 print('Available software specs configuration:', wml_client.software_specifications.ConfigurationMetaNames.get())
 wml_client.software_specifications.list()

 asset_id = 'undefined'
 pe_asset_id = 'undefined'

Create the metadata for package extensions to add to the base specification.

 pe_metadata = {
     wml_client.package_extensions.ConfigurationMetaNames.NAME:
         'My custom library',
     # optional:
     # wml_client.software_specifications.ConfigurationMetaNames.DESCRIPTION:
     wml_client.package_extensions.ConfigurationMetaNames.TYPE:
         'conda_yml'
 }

Create a yaml file containing the list of additional packages and save it as customlibrary.yaml.

Example yaml file:
```
 name: add-regex-package
 dependencies:
     - regex
```
For information on the proper yaml file structure, refer to Examples of customizations.

Store package extension information.

 pe_asset_details = wml_client.package_extensions.store(
     meta_props=pe_metadata,
     file_path='customlibrary.yaml'
 )
 pe_asset_id = wml_client.package_extensions.get_id(pe_asset_details)

Create the metadata for the software specification and store the software specification.

 # Get the id of the base software specification
 base_id = wml_client.software_specifications.get_id_by_name('default_py3.9')

 # create the metadata for software specs
 ss_metadata = {
     wml_client.software_specifications.ConfigurationMetaNames.NAME:
         'Python 3.9 with pre-installed ML package',
     wml_client.software_specifications.ConfigurationMetaNames.DESCRIPTION:
         'Adding some custom libraries like regex', # optional
     wml_client.software_specifications.ConfigurationMetaNames.BASE_SOFTWARE_SPECIFICATION:
         {'guid': base_id},
     wml_client.software_specifications.ConfigurationMetaNames.PACKAGE_EXTENSIONS:
         [{'guid': pe_asset_id}]
 }

 # store the software spec
 ss_asset_details = wml_client.software_specifications.store(meta_props=ss_metadata)

 # get the id of the new asset
 asset_id = wml_client.software_specifications.get_id(ss_asset_details)

 # view new software specification details
 import pprint as pp

 ss_asset_details = wml_client.software_specifications.get_details(asset_id)
 print('Package extensions', pp.pformat(
     ss_asset_details['entity']['software_specification']['package_extensions']
 ))

Propagating software specification and package extension from projects to deployment spaces

You can export custom software specifications and package extensions created in a Watson Studio project to a deployment space using the "Promote" option in the Watson Studio interface. If you want to update the software specification and package extension after they have been promoted to the deployment space, follow these steps.

In the deployment space, delete the software spec, package extensions, associated models(optional) using the Watson Machine Learning python client.
In a Watson Studio project, promote the model, function, or script that is associated with the changed custom software specification and package extension to the space.

Parent topic: Managing frameworks and software specifications