How to to run Continuous Delivery Pipeline stages outside of the IBM public infrastructure using Private Workers.

We are pleased to announce the capability to run Continuous Delivery Pipeline stages outside of the IBM public infrastructure using Private Workers.

Workers are the entities that run the stages of our pipelines, and, up until now, there has been only one means of executing those stages—by using public workers on IBM-managed shared public infrastructure.

With the release of the Private Workers, toolchains can be enhanced with a new private worker tool integration that allows pipeline stages to be configured to run on external Kubernetes environments. Supported platforms include the IBM Cloud Kubernetes Service, IBM Cloud Private, Docker on Desktop, or Red Hat OpenShift.

Why would you want to use private workers?

There are several situations where private workers overcome the constraints of public continuous delivery pipeline workers. Obvious examples are the need for data from a private database or source repository that is not available publicly, the requirement for executing a long-running task, or to adhere to a security policy where jobs need to be run on non-public environments.

In larger organisations, administrators can pre-set private workers on required Kubernetes clusters and make them available for development teams to use.

How do you configure Private Workers?

Private Worker functionality can be typically be enabled in just 30 minutes with the following three steps:

1. Add the Delivery Pipeline Private Worker tool integration to your toolchain.
2. Configure your cluster with a private worker.
3. Choose a private worker in your pipeline stage.

1. Tool integration

A Delivery Pipeline Private Worker tool integration provides the means by which a private worker is made available to pipelines in a toolchain. It provides access to the list of available private workers and to information about them such as status and version number. It also provides the instructions on how to configure a Kubernetes cluster to support private workers.

Create a new tool integration, give it a name, and use the Create ⊕ button to create a ServiceID and APIKey for yourself. Fill in the name and description on the pop-up dialog and click the Create button. Then, click the Create Integration button to add the private worker tool to your toolchain.

More information on ServiceIDs and API Keys.

2. Cluster configuration

You now need to configure your cluster. Note: This step requires access to your cluster via the CLI to execute some kubectl commands. This needs to be done by someone with admin-level access but it is relatively simple; for the purposes of this post, I used a free IBM Cloud Kubernetes Service cluster to which I have an admin account.

You can click on the Private Worker Tool Integration tile on your Toolchains overview page to get the steps required and simply use the copy button to get the commands to run on your cluster.

Once you’ve done this on the cluster, you can confirm that the new worker is ready to accept requests by going to the Tool Integration > Overview page.

3. Pipeline configuration

We’re almost there. Your new private worker is now available to do some work. You just need to configure a stage in your pipeline to use the worker.

You access this via the new Workers tab on the Stage Configuration page.

All that is left is to run the stage and look at the output in the logs on the Stage History page. For the purposes of this example, I added a simple Test Job to my stage that outputs the environment variables, and the log output shows the worker that ran my job.

So, there you have it. We set-up our toolchain, cluster, and pipeline and successfully ran a stage on that cluster—all outside of the IBM public shared infrastructure—by using the new Delivery Pipeline Private Worker feature. Congratulations!

Watch the demo

Watch this short video for a demo of putting private workers to work on an IBM Cloud Kubernetes Service cluster:

Contact us

Now you can try it out yourself. For more information, see our documentation. If you have questions, engage our team via Slack by registering here and join the discussion in the #ask-your-question channel on our public Cloud DevOps @ IBM Slack. We’d love to hear your feedback.