Red Hat OpenShift and sigstore: A powerful pairing for software trustworthiness

As containerized software delivery progresses, Red Hat’s OpenShift has emerged as a beacon of trust in the open source domain. At the heart of this is OpenShift's strategic incorporation of Tekton Chains—a Kubernetes Custom Resource Definition (CRD) controller—to boost supply chain security for nearly every task and pipeline. 

Moreover, OpenShift integrates Cosign extensively, a pivotal tool from the Sigstore family. With Cosign, OpenShift validates that container images are not only built with integrity but are also cryptographically signed, thereby providing an authentication mechanism that asserts the provenance and integrity of the container images. These verifiable signatures offer an additional layer of trust, verifying that no unauthorized changes have been made post-signing.

The synergy of Tekton Chains and Cosign, along with the related tools of Syft and Grype within OpenShift, underscores Red Hat's commitment to driving greater trust and strengthened security in software delivery mechanisms in the modern digital era.

Enhancing image security with Red Hat OpenShift and Cosign

Tekton Chains allows you to manage your supply chain security in Tekton. In its default mode of operation, Chains works by observing all TaskRuns executions in your cluster. Another essential feature is that it can sign OCI registry images with cryptographic key types and services. We will be using Quay as our OCI registry.

Tekton has a few options available to sign your images:

• x509
• Cosign
• KMS
• EXPERIMENTAL: Keyless signing.

The following example demonstrates how to use Cosign to sign your image. You can find a sample pipeline on Github.

To start signing things in Chains, you must generate a keypair and instruct Chains to sign with it via an OpenShift secret. If the key is encrypted, chains expect a private key and password to exist in a Kubernetes secret signing-secrets in the openshift-pipelines namespace.

Step 1. Install the Cosign CLI by following these instructions.

Step 2. Login into your OpenShift instance using the oc client on your terminal.

oc login -u <admin> -p <password> <ocp-api-endpoint>

Step 3. Once logged in, run the following command to create a secret called signing-secrets in the openshift-pipelines namespace which Chains will use to sign your images:

COSIGN_PASSWORD=openshift cosign generate-key-pair k8s://openshift-pipelines/signing-secrets

Step 4. You'll also observe a cosign.pub file generated in the present directory. This file will be utilized later to verify the signed image.

Step 5. Configure Tekton Chains by running the following commands:

cat <<EOF >> chains.yaml
apiVersion: operator.tekton.dev/v1alpha1
kind: TektonChain
metadata:
  name: chain
spec:
  artifacts.oci.storage: oci
  artifacts.taskrun.format: in-toto
  artifacts.taskrun.storage: oci
  config: {}
  targetNamespace: openshift-pipelines
  transparency.enabled: true
EOF
oc apply -f chains.yaml -n <pipelines-namespace>

Step 6. Run the following command a few times to check if the deployment is available:

oc get deployment tekton-chains-controller -n <pipelines-namespace>

Step 7. Wait for the AVAILABLE column to display 1

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
tekton-chains-controller   1/1          1                       1                    106m

Step 8. For Tekton Chains to sign an image, these TaskRun results are required:

• *IMAGE_URL – The URL to the built OCI image
• *IMAGE_DIGEST – The Digest of the built OCI image

The asterisk (*) indicates any expression. For example, if both XXX_IMAGE_URL AND XXX_IMAGE_DIGEST are correctly formatted to point to an OCI image, then chains will pick up on it and try to sign the image.

To get more detail about the process, read this documentation.

The buildah ClusterTask is an example of a pipeline task used to build a container image. It also conveniently sets these results in its TaskRun, which Tekton Chains can then use to sign the image. To create and sign your new image, use this task.

Step 9. You can use Red Hat Advanced Cluster Security for Kubernetes to confirm the integrity of the container images in your clusters by verifying image signatures against preconfigured keys.

You can also create policies to block unsigned images and images that do not have a verified signature and then enforce the policy using the Red Hat Advanced Cluster Security admission controller to stop unauthorized deployment creation.

To verify your image signature in Red Hat Advanced Cluster Security, follow these instructions.

Use the contents of the cosign.pub file created above for your signature integration.

You're also required to provide Red Hat Advanced Cluster Security access to your container registry to scan your image. More details can be found here.

Step 10. Once the policy is set up, create a pipeline task to invoke the roxctl command on your new container image:

./roxctl --insecure-skip-tls-verify image check --endpoint <acs-route> --image <container-image-repo>:<tag>

Step 11. Your pipeline now clones your source code, builds your container image, and runs an ACS check on your image. The pipeline is: clone-source -> build-sign-image -> acs-image-check. 

Step 12. Run your pipeline.

Step 13. Tekton Chains creates annotations on your TaskRun when signing your images. This confirms that your image was signed during this step in your pipeline. The annotations to look out for are chains.tekton.dev/signed and chains.tekton.dev/transparency your signed image:

cosign verify --key cosign.pub <container-image-repo>:<tag>

You can also log into Quay to confirm your image was indeed signed. Your image tag should have a black shield indicating your image was signed using Cosign:

Wrapping up the security conversation

The deeper you delve into the nuances of container security, the clearer it becomes that Red Hat is one of the leaders in fortifying containerized software delivery. This installment follows our initial discussion on the broad approach to containerization security. In that post, we highlighted the pivotal role of Red Hat Advanced Cluster Security (ACS) and Cosign policy. The policy confirms that ACS checks and guarantees that each image has been signed, reinforcing trust at every layer. 

Stay tuned as we continue to unpack the latest advancements and strategies in open source security. Your journey with Red Hat is a testament to our shared commitment to building a safer, more secure digital landscape.