Tutorial: Adding a custom function

In this tutorial, you are writing a custom function that you package and store in a GitHub repository. You are testing the function in your local environment before you register it with the catalog and use it in your calculations.

About this task

In this tutorial, you are writing a custom function, MultiplyByFactor, that multiples an input item by a factor value. As part of the function code, define how the UI controls for input and output items are represented on the configuration UI in Analytics Service. Later, specify the input item and the factor value through the configuration user interface.

Use the execute_local_test method to test the function. You use this method to test any function that is derived from base classes in IoT Functions. The method generates sample data for a function locally. It writes the results to a file named df_test_entitity_for_function_name in the working directory. If the function works, register it with Analytics Service.

Finally, create a sample robot device type in Analytics Service and apply the new MultiplyByFactor function to the device type.

This tutorial uses the Pycharm IDE to create the Python package for the custom function. If you use another IDE or Watson Studio, make sure to use Python 3.x with it.

Important

Alternatively, separate from this tutorial, a custom function starter package is available in IoT functions to help you get started with building your own custom functions.

Before you begin

You can clone a starter package and use it as a sample package in your local environment. Cloning a starter package provides you with an easy way to install the required packages, including IoT Functions, for your custom function. The starter package provides an example of the required directory structure for your custom package.

  1. Install Python 3.x in your environment.
  2. Install GIT.
  3. Install pip3 package manager for Python.
  4. Clone the HelloWorld starter package
    1. Open a terminal window.
    2. Make a new project folder for the starter package. For example, mkdir project.
    3. Change to the project directory. For example, cd project.
    4. Clone the starter package. Enter: git clone --branch starter_package https://github.com/ibm-watson-iot/functions.git
    5. Verify that you cloned the starter package to the project directory.
      1. Change to the functions directory in the starter package. Enter: cd functions
      2. Enter: git status
    6. Copy the working directory path for later use. Enter pwd and copy the path.
  5. Install the PyCharm community edition.
  6. In PyCharm, point your environment to the starter package you cloned. Click Open and browse to find the directory you cloned.
  7. Set up a new Pycharm virtual environment.
    1. Go to PyCharm > Preferences > Project > Project Interpreter.
    2. If you are setting up a new virtual environment:
      1. Click the settings icon and click Add.
      2. Specify the location of your project folder.
      3. Set the base interpreter to your Python 3.x directory.
    3. If you have an existing virtual environment, set the project interpreter to Python 3.x.
  8. Install an editor, such as Xcode, to make code updates.
  9. Install IoT Functions.

Make sure that you are always using the latest version of IoT Functions. Check the version of each Python module that is supported by Analytics Service in the Python module dependencies topic. If you do not use the exact version that is supported, your pipeline might break.

Steps

Complete these steps to apply a calculation to sample data by using a custom function.

Step 1: Define your calculation

Acme is a company that develops robotic arms. The company is introducing a new robotic arm and is testing that it performs as well as the existing robotic. In early testing, the team at Acme discovered that for some of the robots, the distance traveled during testing was too high. After investigation, the team discovered that the tools used to do the testing were adding a delay to the speed and travel times of the robot. The operations manager wants to adjust both values by a factor of 2 but wants the flexibility to change that value later.

The analyst identifies that a similar calculation might be useful in other calculations and generalizes the function by writing a MultiplyByFactor class.

distance * factor

Step 2: Package your function and store it in GitHub.

  1. Create a new repository on GitHub for the starter package you cloned.
    1. The repository must have a URL that is accessible from a pip install.
    2. A best practice is to make the repository private. The repository can be accessed using a personal access token.
  2. Add a file to the repository to create the master branch. For example, add an empty file that is named test.md and click Commit new file.
  3. Open the project in PyCharm locally for editing.
  4. Verify that the project has the following directory structure and files:
    functions
|
|__ setup.py
|
|__ scripts
  |
  |_ local_test_of_function.py
|
|__ custom
  |
  |_ functions.py
  |_  __init__.py
  5. Change the module name, custom, to a unique name, such as customyourinitials. For example, customld.

  6. Verify that the contents of the setup.py file includes the following code: Remember: Change the module name to match your module name.

    from setuptools import setup, find_packages

setup(name='customld', version='0.0.1', packages=find_packages(),
      install_requires=['iotfunctions@git+https://github.com/ibm-watson-iot/functions.git@production'])

  7. Create a multiplybyfactor<yourinitials>.py function module in custom<yourinitials> and paste the following code. Update the variables in <> to match your environment. For example, you might set PACKAGE_URL to 'git+https://<XXXXXX>@github.com/github.com/jones/starter@starter_package'. Note: In this function, you implement two methods. You add your calculation to the execute method. You set the inputs and outputs arguments for the function in the build_ui method. These arguments are configured through the configuration user interface.

    import inspect
import logging
import datetime as dt
import math
from sqlalchemy.sql.sqltypes import TIMESTAMP,VARCHAR
import numpy as np
import pandas as pd

from iotfunctions.base import BaseTransformer
from iotfunctions import ui

logger = logging.getLogger(__name__)

# Specify the URL to your package here.
# This URL must be accessible via pip install.
# Example assumes the repository is private.
# Replace XXXXXX with your personal access token.
# After @ you must specify a branch.

PACKAGE_URL = 'git+https://XXXXXX@github.com/<user_id><path_to_repository>@starter_package'

class MultiplyByFactor<YourInitials>(BaseTransformer):

    def __init__(self, input_items, factor, output_items):

        self.input_items = input_items
        self.output_items = output_items
        self.factor = float(factor)
        super().__init__()
    def execute(self, df):
        df = df.copy()
        for i,input_item in enumerate(self.input_items):
            df[self.output_items[i]] = df[input_item] * self.factor
        return df

    @classmethod
    def build_ui(cls):
        #define arguments that behave as function inputs
        inputs = []
        inputs.append(ui.UIMultiItem(
                name = 'input_items',
                datatype=float,
                description = "Data items adjust",
                output_item = 'output_items',
                is_output_datatype_derived = True)
                      )        
        inputs.append(ui.UISingle(
                name = 'factor',
                datatype=float)
                      )
        outputs = []
        return (inputs,outputs)

Step 3: Save your credentials to a file

Set credentials to connect to Analytics Service.

  1. Create a credentials_as.json file in the scripts folder in your working directory. On the user interface, go to the Services tab.
  2. Select Watson IoT Platform Analytics and click View Details.
  3. In the Environment Variables field, click Copy to Clipboard.
  4. Paste the contents of the clipboard into the credentials_as.json file.

Important

The credentials file is used to run or test the function locally. Do not push this file to your external repository in step 4.

Step 4: Push your local changes to your GitHub repository

Note:

Push your function code to your external repository in GitHub.

  1. Open a terminal window in the project folder.
  2. Enter: git remote -v. The remote repository is still pointing to https://github.com/ibm-watson-iot/functions
  3. Change the remote origin URL by using the following commands:
     git remote set-url origin URL_To_YOUR_GITHUB_REPOSITORY
    Confirm that the Fetch and Push URLs are pointing to your repository.
  4. Add the upstream remote by using the following command. Enter:
     git remote add upstream https://github.com/ibm-watson-iot/functions.git
  5. Add your files to GIT and commit them.
    1. In PyCharm, select the custom<your_initials> directory.
    2. Select Git > Add.
    3. Select Git > Commit.
    4. In the Commit Changes window, click Commit.
    5. Select Git > Repository > Push.
    6. Click Push.
  6. Verify that your module was pushed to a starter_package branch in your repository.

Step 5: Install the custom function in your local environment

Install the custom function in your local environment. Update the variables in <> to match your environment. 

pip3 install git+https://<XXXXXX>@github.com/<user_id><path_to_repository>@starter_package --upgrade

For example:

pip3 install git+https://<XXXXXX>@github.com/jones/starter@starter_package --upgrade

Replace with your personal access token.

Step 6: Test your custom function locally.

  1. Create a script called test_my_custom_function.py in the scripts folder. In the script, import Python libraries and packages:
    import datetime as dt
import json
import pandas as pd
import numpy as np
from sqlalchemy import Column, Integer, String, Float, DateTime, Boolean, func
from iotfunctions.base import BaseTransformer
from iotfunctions.metadata import EntityType
from iotfunctions.db import Database
from iotfunctions import ui
  2. Connect to Analytics Service. In the test_my_custom_function.py script, add:
    with open('credentials_as.json', encoding='utf-8') as F:
  credentials = json.loads(F.read())
db_schema = None
db = Database(credentials=credentials)
  3. Import and instantiate the function. In the test_my_custom_function.py script, add the following code. Update the variables in <> to match your environment.
    from custom<yourinitials>.multiplybyfactor<yourinitials> import MultiplyByFactor<YourInitials>
fn = MultiplyByFactor<YourInitials>(
    input_items = ['speed', 'travel_time'],
    factor = '2',
    output_items = ['adjusted_speed', 'adjusted_travel_time']
              )
df = fn.execute_local_test(db=db, db_schema=db_schema, generate_days=1,to_csv=True)
print(df)
  4. Run the script from the command line. The date frame results are saved to a CSV file. Look for df_test_entity_for_multiplybyfactor<your_initials>.csv in the scripts directory. Enter:
    python3 test_my_custom_function.py

Step 7: Register your custom function.

  1. Create a script called register_my_custom_function.py in the scripts folder. In the script, import Python libraries and packages:
    import datetime as dt
import json
import pandas as pd
import numpy as np
from sqlalchemy import Column, Integer, String, Float, DateTime, Boolean, func
from iotfunctions.base import BaseTransformer
from iotfunctions.metadata import EntityType
from iotfunctions.db import Database
from iotfunctions import ui
  2. Connect to Analytics Service. In the register_my_custom_function.py script, add:
    with open('credentials_as.json', encoding='utf-8') as F:
  credentials = json.loads(F.read())
db_schema = None
db = Database(credentials=credentials)

  3. In the register_my_custom_function.py script, to register the function, add the following code. Update the variables in <> to match your environment. 

    from custom<yourinitials>.multiplybyfactor<yourinitials> import MultiplyByFactor<YourInitials>

db.register_functions([MultiplyByFactor<YourInitials>])
  4. Run the script from the command line. For example:
    python3 register_my_custom_function.py

Step 8: Create a sample device type.

  1. On the Setup tab, from the Devices tab, click Screenshot of the Create new data item button. to add a new device type.
  2. In the Add device type dialog, on the Select a template page, select the Robot Sample Type template.
  3. Assign a name to the device type.
  4. Click Create.
  5. Select the sample device type, click Manage device type, and then click Data.

KPI data takes up to 5 minutes to generate.

Step 9: Apply the custom function to the device type.

  1. If you have not created a distance metric, add it. Follow the steps in Tutorial: Adding expressions
  2. From the Data tab, click Screenshot of the Create new data item button..
  3. Select the MultiplyByFactor<YourInitials> function from the catalog.
  4. In the factor field, enter 2.
  5. In the input_item field, select distance.
  6. Click Next.
  7. Rename the output 'adjusted_distance'.
  8. Click Create.
  9. In the data item list, select adjusted_distance. Wait up to 5 minutes for Analytics Service to evaluate the custom function against the sample data.