IBM Cloud Continuous Delivery has introduced support for IBM Cloud Secrets Manager in CI/CD toolchains.

Does the prospect of a security breach in your CI/CD processes keep you up at night? Is your DevOps team battling to achieve regulatory compliance? Is your development team sharing secrets over Slack, post-it notes or text files? Does your product include hard-coded secrets within source code or files? Are secret values committed into Git repos? Are you unable to grant or revoke access to your secrets in a managed way? Is it impossible for your organisation to rotate secrets in a repeatable way without breaking CI/CD processes due to secret proliferation and sprawl?

If the answer is yes to any of these questions, then you’re not alone in the DevOps space. However, if you are serious about taking security to the next level, then the IBM Cloud Continuous Delivery service has tools to help you minimise your risk and vulnerability exposure.

IBM Cloud Continuous Delivery has introduced support for IBM Cloud Secrets Manager in CI/CD toolchains and pipelines to further bolster its secrets management tool integration capabilities. That’s right, in addition to the existing secrets management tool integration support for IBM Key Protect and HashiCorp Vault, you can make use of the Secrets Manager tool integration for secrets in your CI/CD toolchains and pipelines.

DevSecOps in CI/CD using Shift Left

DevSecOps should be a day-to-day practice amongst your development team to integrate security into CI/CD development and delivery processes. One such measure in DevSecOps that is universally accepted as IBM Cloud best-practice is the ethos of the “Shift Left” mantra. Shift Left encourages everyone involved in the software development lifecycle — from software developers to operational engineers — to incorporate security controls and measures from the right side of the development lifecycle (end) through to the left side of the development lifecycle (beginning).

An organization that utilises DevSecOps brings together their security subject matter experts as an integral part of the development team, where their task is to ensure all resources in the software and CI/CD process are secure and auditable. As a whole, the development team proactively thinks about automating security into the development process and product in a security-conscious manner. A Shift Left mantra across your development team allows security risks and exposures to be identified early and ensures that potential security threats and vulnerabilities are accounted for and addressed before later development or deployment phases.

CI/CD and IBM Cloud Secrets Manager

By using the Secrets Manager tool integration, you can reference Arbitrary-type secrets in your CI/CD toolchains and pipelines. Your sensitive and confidential secrets are securely stored and managed by your organisation’s administrator within a single-tenant IBM Cloud Secrets Manager instance. Secrets stored in IBM Cloud Secrets Manager are always dynamically resolved By-reference from within your DevOps toolchains and pipelines. Therefore, enabling actions like granting or revoking access or rotating secret values to be performed at a centralised point that then takes effect across all pipelines with references to those secrets. In addition, access is governed by IBM Cloud Identity and Access Management (IAM) to ensure access is only ever granted to a genuine IBM trusted and authorised user or authorised service.

Note: “By-reference” is an integral mechanism within the CI/CD toolchain and pipeline framework that ensures secret values are not used by your CI/CD toolchains, pipelines or IBM Cloud DevOps web console user interface. A safe, secure “vault” reference is used instead of the actual raw secret value. The actual arbitrary raw secret value is resolved dynamically on the server-side based on the set of attributes in the secret reference and associated tool integration. Then, after access is granted by IAM, the secret value is used at the point-in-time it is required in your CI/CD workflow. In addition, it is obfuscated from pipeline logs and echo attempts from scripts; however, other uses of secret values within custom scripting or persistence within a pipeline remain under the control and management of the pipeline developer.

By using the IBM Cloud Secrets Manager (or indeed IBM Key Protect or HashiCorp Vault) tool integrations for secrets in your CI/CD toolchains and pipelines, you are removing the need to hard-code or copy/paste secrets like API Keys or passwords directly into your CI/CD resources. Instead, special vault secret references are used to resolve your sensitive data. By cutting off this drip feeding of raw secret values into your CI/CD toolchains and pipelines, you are taking one big step in applying Shift Left practice.

Using IBM Cloud Secrets Manager in IBM Cloud DevOps setup

Let’s consider a typical best-practice use case where a Secrets Manager service instance has been provisioned by your organisation’s administrator. We’ll call it fsapp-secrets-manager for the purposes of this example. You and a number of your development team colleagues have been granted permission to list secrets within it but no permission to read them. In addition, you are in an Access Group that is granted permission to create IAM Authorisations between the toolchain and Secrets Manager services.

You now have a requirement to create a pipeline based on a particular toolchain template, and this pipeline can only be deployed using a specific API Key secret value that you don’t have, currently. Fortunately, you don’t need to know it because your organisation’s administrator has already saved the API Key secret into the fsapp-secrets-manager instance as an Arbitrary type secret called fsapp-pipeline-apikey.  Remember, this is the Secrets Manager service instance from which you have permission to list secrets but no permission to read the actual secret values. Therefore, you continue to load the specific toolchain template into the IBM Cloud DevOps web console and proceed to configure a Secrets Manager tool integration using the Secrets dialog as seen in Figure 1:

After you have configured the region and resource group and selected the fsapp-secrets-manager instance name, you then select the Default secret group followed by the required fsapp-pipeline-apikey secret from the list of secret names in the Secrets dialog (also seen in Figure 1). Importantly, you note the fact that after selecting the required secret from the secret-name dropdown menu and clicking on the OK button, a new Secrets Manager tool integration has been added to your toolchain. This new tool integration has its attributes already configured based on the values for region, resource group and instance name that you selected in the Secrets dialog to begin with. In addition, an auto-generated name has been set on the tool integration for your convenience with the value ibm-secrets-manager-1, as seen in Figure 2:

You also note the fact that instead of a raw secret value now being present in the secure API Key input field, a purple-coloured pill with the selected group and secret name has been injected, as seen in Figure 3:

To satisfy your curiosity, you also inspect the underlying browser DOM, thinking the raw secret value might be hidden in the DHTML of the underlying IBM Cloud DevOps Setup page, but you are immediately reassured to only to find a vault reference containing the following value in the secure API Key input field: “{vault::ibm-secrets-manager-1.Default.fscapp-pipeline-apikey}”

You then proceed to click on the Create toolchain button where a service Authorization is now automatically created by IAM on your behalf within the organisation account between this newly created toolchain service instance and selected Secrets Manager service instance. This Authorization declares that this toolchain service instance can list secrets using a Viewer role and also has permission to read secret values in the associated Secrets Manager instance by way of a SecretsReader role.

This is an important role that includes the read permission in order to enable your newly created toolchain to resolve the injected vault reference by reading the associated raw secret value from the Secrets Manager instance dynamically and securely during a pipeline run.

The above example use case captures the essence of applying a Shift Left practice right at the very beginning of a CI/CD toolchain creation process — even before a corresponding pipeline is instantiated — and therefore preventing leakage of raw secret values into the development and deployment pipeline environment.

Using IBM Cloud Secrets Manager in your toolchain templates

It is also worth mentioning here that the above described toolchain template in the example use case could be enhanced to include the required Secrets Manager tool integration definition and vault secret reference.  Therefore, upon loading the toolchain template into the IBM Cloud DevOps web console, the required fsapp-pipeline-apikey secret would be automatically selected and validated by the DevOps web console on behalf of the currently logged-in user. Consider the following toolchain.yml fragment from an enhanced version of the toolchain template where you can see the Secrets Manager ibm-secrets-manager-1 service definition and associated API Key input field with a hard-coded vault reference to the fsapp-pipeline-apikey secret:

services:

  ibm-secrets-manager-1:
    service_id: secretsmanager
    parameters:
      name: ibm-secrets-manager-1
      region: us-south
      resource-group: Default
      instance-name: fsapp-secrets-manager

form:
  pipeline:
    parameters:
      api-key: "{vault::ibm-secrets-manager-1.Default.fsapp-pipeline-apikey}"
    schema:
      $ref: deploy.json

Note: Importantly, this enhanced version of the toolchain template definition can be safely committed into a source code configuration management system like a Git repo (or similar) as no sensitive secrets are included.

How is Secrets Manager different from Key Protect?

There are a few key differences between using Key Protect and Secrets Manager to store your sensitive data. Secrets Manager offers flexibility with the types of secrets that you can create and lease to applications and services on-demand. Key Protect, on the other hand, delivers on encryption keys that are rooted in FIPS 140-2 Level 3 hardware security modules (HSMs). Continuous Delivery supports the Standard secret type in Key Protect. This is similar to the Arbitrary secret type in Secrets Manager. However, Secrets Manager offers additional capabilities for this generic type of secret that are useful in the context of CI/CD processes.

For more information, see Managing secrets in IBM Cloud.

How is Secrets Manager different from HashiCorp Vault?

With Secrets Manager, you can centrally manage secrets for your services or apps in a dedicated, single-tenant instance. To control who on your team has access to specific secrets, you can create secret groups that map to Identity and Access Management (IAM) access policies in your IBM Cloud account. And, you can use IBM Cloud Activity Tracker to track how users and applications interact with your Secrets Manager instance.

Conclusion

The inherent benefits from applying the approach of using a Secrets Manager tool integration and underlying Secrets Manager service instance are pretty profound. It is true to say that the same benefits are also realisable by leveraging the IBM Key Protect or HashiCorp Vault tool integrations in CI/CD, but it is noteworthy to mention that IBM Cloud Secrets Manager does provide a rich feature set that lends itself well to CI/CD and the DevSecOps workflows where numerous and varied arbitrary secrets are required. In particular, it is tightly coupled with IAM, whereby additional IAM Access Policy and Secret Group mappings can be configured, amongst many other options that include support for secret types beyond Arbitrary, with options for IAM Credential, and Username/Password secret types, Secret rotation and Expiry, and Bring Your Own Key (BYOK) integration with IBM Key Protect to envelope encrypt all secrets in a service instance using your own encryption key.

By using the IBM Cloud Secrets Manager, IBM Key Protect or HashiCorp Vault tool integrations for secrets in IBM Cloud Continuous Delivery, you are ensuring that your DevOps team and CI/CD processes are securely accessing and referencing secrets stored in these industry-standard, enterprise-ready secrets management providers in a dynamic and ephemeral manner. Therefore, you up your game as a DevSecOps advocate and proponent and fend off risk and vulnerability whilst helping to keep regulatory compliance in check.

I invite you to try out the IBM Cloud Secrets Manager tool integration in your CI/CD toolchains and pipelines today at IBM Cloud DevOps, where you can adopt it in applying your own DevSecOps best practices.

Get started

• Tutorial to get started with Toolchains.
• Learn more about IBM Cloud Continuous Delivery.
• Learn more about IBM Cloud Secrets Manager.
• If you have any questions, get help directly from the IBM Cloud development teams by joining us on Slack.