Amazon SageMaker frameworks

You can use Amazon SageMaker to log payload, feedback, and to measure performance accuracy, bias detection, explainability, and auto-debias function in IBM Watson OpenScale.

IBM Watson OpenScale fully supports the following Amazon SageMaker frameworks:

Table 1. Framework support details

Framework support details
Framework Problem type Data type
Native Classification Structured
Native Regression1 Structured

1Support for regression models does not include drift magnitude.

Adding Amazon SageMaker to Watson OpenScale

You can configure Watson OpenScale to work with Amazon SageMaker by using one of the following methods:

Sample Notebooks

The following Notebooks show how to work with Amazon SageMaker:

Specifying an Amazon SageMaker ML service instance

Your first step in the Watson OpenScale tool is to specify an Amazon SageMaker service instance. Your Amazon SageMaker service instance is where you store your AI models and deployments.

Connect your Amazon SageMaker service instance

Watson OpenScale connects to AI models and deployments in an Amazon SageMaker service instance. To connect your service to Watson OpenScale, go to the Configure The configuration tab icon tab, add a machine learning provider, and click the Edit The configuration tab icon displays. icon. In addition to a name and description and whether the environment Pre-production or Production, you must provide the following information that is specific to this type of service instance:

You are now ready to select deployed models and configure your monitors. Watson OpenScale lists your deployed models on the Insights dashboard where you can click Add to dashboard. Select the deployments that you want to monitor and click Configure.

For more information, see Configure monitors.

Payload logging with the Amazon SageMaker machine learning engine

Add your Amazon SageMaker machine learning engine

A non-IBM Watson Machine Learning engine is bound as Custom by using metadata. No direct integration with the non-IBM Watson Machine Learning service is possible.

SAGEMAKER_ENGINE_CREDENTIALS = {
                   'access_key_id':””,
                   'secret_access_key':””,
                   'region': '}

wos_client.service_providers.add(
        name="AWS",
        description="AWS Service Provider",
        service_type=ServiceTypes.AMAZON_SAGEMAKER,
        credentials=SageMakerCredentials(
            access_key_id=SAGEMAKER_ENGINE_CREDENTIALS['access_key_id'],
            secret_access_key=SAGEMAKER_ENGINE_CREDENTIALS['secret_access_key'],
            region=SAGEMAKER_ENGINE_CREDENTIALS['region']
        ),
        background_mode=False
    ).result

To see your service subscription, run the following code:

client.service_providers.list()

SageMaker ML binding

Add Amazon SageMaker ML subscription

To add the subscription, run the following code:

asset_deployment_details = wos_client.service_providers.list_assets(data_mart_id=data_mart_id, service_provider_id=service_provider_id).result
asset_deployment_details

deployment_id='684e35eee8a479470cee05983e1f9d64'
for model_asset_details in asset_deployment_details['resources']:
    if model_asset_details['metadata']['guid']==deployment_id:
        break


aws_asset = Asset(
        asset_id=model_asset_details['entity']['asset']['asset_id'],
        name=model_asset_details['entity']['asset']['name'],
        url=model_asset_details['entity']['asset']['url'],
        asset_type=model_asset_details['entity']['asset']['asset_type'] if 'asset_type' in model_asset_details['entity']['asset'] else 'model',
        problem_type=ProblemType.BINARY_CLASSIFICATION,
        input_data_type=InputDataType.STRUCTURED,
    )

from ibm_watson_openscale.base_classes.watson_open_scale_v2 import ScoringEndpointRequest
deployment_scoring_endpoint = model_asset_details['entity']['scoring_endpoint']
scoring_endpoint = ScoringEndpointRequest(url = model_asset_details['entity']['scoring_endpoint']['url'] )

deployment = AssetDeploymentRequest(
        deployment_id=model_asset_details['metadata']['guid'],
        url=model_asset_details['metadata']['url'],
        name=model_asset_details['entity']['name'],
        #description="asset['entity']['description']",
        deployment_type=model_asset_details['entity']['type'],
        scoring_endpoint =  scoring_endpoint
    )

asset_properties = AssetPropertiesRequest(
        label_column="Risk",
        prediction_field='predicted_label',
        probability_fields=['score'],
        training_data_reference=training_data_reference,
        training_data_schema=None,
        input_data_schema=None,
        output_data_schema=None,
        feature_fields=feature_columns,
        categorical_fields=categorical_columns
    )

subscription_details = wos_client.subscriptions.add(
        data_mart_id=data_mart_id,
        service_provider_id=service_provider_id,
        asset=aws_asset,
        deployment=deployment,
        asset_properties=asset_properties,
        background_mode=False
).result

To get the subscription list, run the following command:

subscription_id = subscription_details.metadata.id
subscription_id

details: wos_client.subscriptions.get(subscription_id).result.to_dict()

Enable payload logging

To enable payload logging, run the following command:

request_data = {'fields': feature_columns, 
                'values': [[payload_values]]}

To get logging details, run the following command:

response_data = {'fields': list(result['predictions'][0]),
                 'values': [list(x.values()) for x in result['predictions']]}

Scoring and payload logging

To store the request and response in the payload logging table, run the following code:

wos_client.data_sets.store_records(data_set_id=payload_data_set_id, request_body=[PayloadRecord(
           scoring_id=str(uuid.uuid4()),
           request=request_data,
           response=response_data,
           response_time=460
)])

For languages other than Python, you can also log payload by using a REST API.

Next steps

Parent topic: Supported machine learning engines, frameworks, and models