How to sign your images and enforce policies to avoid security risks and build trust.

The IBM Cloud for Financial Services is really a combination of cloud-native development best practices around access, isolation, encryption, trust, logging and auditability. It involves network isolation, strict security policies, scanning and more. That is all great by itself, but it means nothing if you can’t trust the workload that you’re deploying into your secured clusters. If you deploy a compromised image into your cluster, then you put the entire system at risk.   

There have been many cases in the real world where entire systems have been breached because compromised images found their way into production clusters. This has included everything from hidden cryptocurrency mining on compromised servers to hackers breaking out of privileged containers and taking over hosts and/or nodes of the cluster.  

One way to ensure trust in your containerized workloads is to use signed images. You can sign images in your DevOps processes, either when a container image is built or when it is pushed to a container registry.  Then, on your clusters, you can use Image Security Enforcement to create policies that enforce those signatures. This means that if there’s an attempt to deploy an image into your cluster and it either isn’t signed or doesn’t have a valid signature matching your policy, that container image will be blocked.   

This process creates trust by ensuring that your container images have not been altered or compromised since they left your build process. By building off of trusted base images and signing your own images, you can have increased trust that unwanted code has not been injected into your deployments.

So, how do you go about signing your images and enforcing policies?  

Create a signing key

First, you need to enable Container Image Security Enforcement on your Kubernetes or OpenShift cluster.  This can be done in the cluster user interface on IBM Cloud or by using the ibmcloud command line utility:  

ibmcloud ks cluster image-security enable -c <cluster name>

This command will configure the cluster to use Container Image Security — which installs Portieris in the cluster — and will also create a ClusterImagePolicy instance to enforce signatures for IBM Cloud images.

Portieris is a Kubernetes admission controller for the enforcement of image security policies. With Portieris, you can create image security policies for each Kubernetes namespace or at the cluster level, and you can enforce different rules for different images. Portieris will block deployment for any image that fails signature validation as defined by the image policies, thus ensuring that the images running inside of your cluster are unmodified from their original source.

Signing images

Now that you’ve got Container Image Security Installed, you need to start using signed images.

The recommended approach for signed images in Kubernetes clusters is to sign images using Red Hat Signatures. This is the same signing process that is used to sign Red Hat RPM packages. 

In a nutshell, the process is as follows:

Create a GPG (Gnu Privacy Guard) key that you can use to sign your images. Be sure to save both the public and private keys for this in a safe place.

Sign the image:

• Do this when building an image by using the buildah command:

  buildah --sign-by <KEY_FINGERPRINT> --storage-driver=overlay push --digestfile ./image-digest ${APP_IMAGE} docker://${APP_IMAGE}

• Alternatively, you can sign when pushing an image or copying between registries using the skopeo command:

  skopeo --sign-by <KEY_FINGERPRINT> copy docker://${IMAGE_FROM} docker://${IMAGE_TO}

Signing Images with the Cloud Native Toolkit

The process described above is a very simplified version of the signing process. In a real-world implementation, you would add signing to your DevOps pipelines, either in a build or image-release task. In the Cloud Native Toolkit, we have have Tekton pipelines and tasks to automate our DevOps process — covering everything from compiling code, running automated test suites, building images, scanning images, etc. — up until triggering a GitOps change that can deploy your latest code into your OpenShift clusters. As part of this process, images are initially built and stored in the internal OpenShift container registry, and once all tests and scans pass, then the image is copied into the IBM Cloud Container Registry.   

We’ve recently added parameters to the Cloud Native Toolkit’s image release task that can sign your images when they are copied from the internal registry into the IBM Cloud Container Registry.    

I mentioned earlier that we need a GPG key to sign the images. The private key is necessary to sign the images, and the public key is used to verify the signatures. You can store both the public and private keys in a vault for safe keeping. The recent modifications to the image release task will pull the private key from a vault. Either IBM Key Protect for IBM Cloud or IBM Cloud Hyper Protect Crypto Services will work because they have an identical API, though Hyper Protect Crypto Services has a higher level of security and is recommended.   

So, the image release task can now pull the private key from your vault of choice (depending how you configure it) and sign the image as it is pushed to the IBM Cloud Container Registry.   

Note: You can learn more about image-signing steps and how to set it up with your OpenShift pipelines.

Enforcing policies

To enforce policies, the public key component of the GPG key needs to be accessible to the cluster. The easiest way to do this is to create a Secret on the cluster that contains the GPG public key:

gpg --export --armour <KEY_FINGERPRINT> > key.pubkey 
oc create secret generic image-signing-public-key --from-file=key=key.pubkey

Finally, you can create image policies to enforce the use of signed images from specific container registries. These policies can be applied globally to the entire cluster using a ClusterImagePolicy or to a specific namespace using an ImagePolicy resource. In those policies, rules can be defined for enforcement for specific container registries/namespaces or globally to all container registries used by the cluster.

For example, the following ClusterImagePolicy enforces a policy that all images in the container registry private.us.icr.io/mynamespace/* must be signed by the public key that was created earlier and placed into the image-signing-public-key cluster secret. This policy should be updated for your own registry namespace and images:

apiVersion: portieris.cloud.ibm.com/v1
kind: ClusterImagePolicy
metadata:
  name: mynamespace-cluster-image-policy
spec:
   repositories:
    - name: "private.us.icr.io/mynamespace/*"
      policy:
        mutateImage: false
        simple:
          requirements:
          - type: "signedBy"
            keySecret: image-signing-public-key

You can find more information about policies and enforcement and image mutation in the Portieris Policies documentation.

What next?

You can learn more about image signing and cloud-native best practices by checking out the following links:

• New Cluster Image Security Enforcement Feature
• Container Image Security Enforcement in the Cloud Native Toolkit
• Container Image Security Enforcement Policies