Securing the deployment of OpenShift Container Platform 4

There have been some changes to the way we deliver OpenShift Container Platform 4 from OpenShift Container Platform 3. The installer and client binaries are delivered via mirror.openshift.com, as well as access.redhat.com. Also, container images are delivered from Quay.io primarily, and most of them are mirrored to the Red Hat container registries such as registry.access.redhat.com and registry.redhat.io. This was done to speed up the rate at which we can deliver updates, and facilitate over-the-air updates. You might be wondering how we ensure the integrity of bits delivered via this new delivery mechanism, so we’re going to dive into that in this post.

Verifying the integrity of the installer and/or client binaries

Starting with mirror.openshift.com where you obtain the installer, and client Golang binaries, you’ll notice a sha256sum.txt document that contains a hash of each of the installer and client binaries hosted there. If you obtain the installer and client binaries from access.redhat.com you’ll still need to get the sha256sum.txt.sig from mirror.openshift.com. On a RHEL host, the key used to sign the sha256sum.txt is located on disk at /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release. 

This is probably the best option for verifying the signature, assuming you trust the channel which you obtained the RHEL bits from. For example, if you provisioned RHEL in AWS EC2 using an official AMI the channel is trustworthy. You can also obtain the official signing key from the Red Hat Product Security section of the customer portal. 

Just be aware that if an attacker has already compromised your machine then the integrity of the official key could be compromised in transit from that page. You could also rely on your web of trust to verify the authenticity of the official Red Hat signing key.

Once you have a copy of the official Red Hat signing key and sha256sum.txt.sig locally, you can use these gpg commands to create a temporary keyring and verify the signature is valid.

$ gpg --no-default-keyring --keyring ./temp.keyring --import /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
gpg: key FD431D51: public key "Red Hat, Inc. (release key 2) <security@redhat.com>" imported
gpg: key 2FA658E0: public key "Red Hat, Inc. (auxiliary key) <security@redhat.com>" imported
gpg: Total number processed: 2
gpg:               imported: 2 (RSA: 1)

$ gpg --no-default-keyring --keyring ./temp.keyring --output sha256sum.txt sha256sum.txt.sig
gpg: Signature made Fri 22 Nov 2019 09:25:33 EST using RSA key ID FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51

Note that /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release contains two public keys. The official ‘release key 2’, and an older ‘auxiliary key’. At the time of writing the OpenShift Container Platform 4 binaries are verified with the ‘release key 2’ key, not the ‘auxiliary key’.

This command outputs sha256sum.txt locally. You can use the sha256 hashes in this file to verify the integrity of the installer, and/or client binary. In order to do that, download the installer, and/or client binaries, calculate a sha256sum locally and check it matches the hash listed in the verified sha256sum.txt file.

Verifying the integrity of the container images

There are two methods to provision the infrastructure used by OpenShift Container Platform 4: Installer Provisioned Infrastructure (IPI), and User Provisioned Infrastructure (UPI). These 2 methods are explained in the installation section of the documentation. We’ll focus on the IPI method because it’s fully automated, and therefore less transparent than UPI. We’ll also use Amazon Web Services (AWS) as an example in this article, but the same concepts apply to other cloud providers or OpenStack which are supported environments for IPI installations.

Establishing a Chain of Trust

The bootstrap machine is provisioned using a reference from the installer binary. “The installation creates an encrypted AMI for the bootstrap and control-plane machines. The encrypted AMI is copied from the RHCOS AMI uploaded by Red Hat to each availability zone. The encryption uses the default EBS key for your target account and region (aws kms describe-key --key-id alias/aws/ebs) from the upstream installer documentation.The bootstrap machine includes a reference to a release image by digest.

Referencing an image by digest maintains integrity of the image contents because the digest is calculated from the image contents, it’s a form of content addressable storage. In fact, if two registries implement the Docker Registry Schema v2 and both host the same container image, the digest will be the same. That makes referring to images by digest an easy way for Red Hat to mirror container registries for resiliency, but also for customers to mirror OpenShift Container Platform images themselves for offline installations, or performance reasons.

The installer and release image refer to images by digest. This makes it harder for an attacker to compromise the image at rest on the registry, or in transit to the bootstrap node. If an attacker was to change the target image in any way, the digest would calculate to a different value from that referred to by the release image. That is why the container-tools library used by OpenShift Container Platform recalculates the digest locally after each pull. 

Imagine a scenario where an attacker sets up a fake registry and intercepts connections from the installer to the official quay.io registry. They could serve ‘fake’ images which don’t use the ‘correct’ digest of their malicious images. If the attacker was able to intercept the network connection and redirect the installer to download their faked, malicious images and we didn’t verify the digest of the image locally, we would be vulnerable to attack. Because OpenShift Container Platform 4 uses the container tools library it throws an error if the hash of the container image doesn’t match the reference from the release image.

Verifying the integrity of an upgrade

During an upgrade, OpenShift Container Platform 4 downloads the target release image and the corresponding signature from an atomic signature store also hosted on mirror.openshift.com. It then checks that the image digest was signed with the official ‘release key 2’. You can verify this manually using a command similar to that used in the install section:

$ wget https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=f28cbabd1227352fe704a00df796a4511880174042dece96233036a10ac61639/signature-1

$ gpg --no-default-keyring --keyring /tmp/keyring --output signature signature-1 
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
gpg: Signature made Thu 28 Nov 2019 04:41:20 AEST
gpg:                using RSA key 199E2F91FD431D51
gpg: Good signature from "Red Hat, Inc. (release key 2) <security@redhat.com>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 567E 347A D004 4ADE 55BA  8A5F 199E 2F91 FD43 1D51

$ cat signature
{"critical": {"image": {"docker-manifest-digest": "sha256:f28cbabd1227352fe704a00df796a4511880174042dece96233036a10ac61639"}, "type": "atomic container signature", "identity": {"docker-reference": "quay.io/openshift-release-dev/ocp-release:4.2.9"}}}

The code that was added to OpenShift here does the manual steps described above automatically for each upgrade. The docker-manifest-digest value in the signed payload is compared against the digest of the target release image and if there is a mismatch the verification fails. 

You can see messages related to signature verification failure in the web console. For example, on a 4.2 OCP cluster, if trying to upgrade to a CI build, it will fail and you should see an error message like this:

$ oc adm upgrade --to-image registry.svc.ci.openshift.org/ocp/release@sha256:0944792c44adaaf4cc3866d9459fcbcc7c3ba3809bb0c926ac4f1bef241c9133 --allow-explicit-upgrade

Updating to release image registry.svc.ci.openshift.org/ocp/release@sha256:0944792c44adaaf4cc3866d9459fcbcc7c3ba3809bb0c926ac4f1bef241c9133

Verifying the integrity of Operator Lifecycle Manager operators

While the release image refers to all the content that makes up a default installation of OpenShift Container Platform 4, it is possible to extend the functionality of OpenShift Container Platform 4 after installation using the Operator Lifecycle Manager. 

There are three different catalogs of content (community, certified, and Red Hat), organized by origin. The Red Hat catalog contains only images from the Red Hat container registry. In order to support mirroring of images for disconnected installation of OpenShift 4 operators need to be referenced by digest. At the time of writing, not all of the Red Hat operators referred to by the OLM are using a digest yet. However, since they are shipped from the Red Hat registry we can use signatures to verify their content. 

Conclusion

Red Hat helps you verify the integrity of OpenShift Container Platform 4 installation and upgrade by using signatures on the installer binaries and release image. Once a release image is downloaded and verified, it refers to the rest of the images which make up the default OpenShift Container Platform 4 using their digest. 

Pulling images by digest and verifying the content matches that digest makes it harder for an attacker to compromise the integrity of the images. The Operator Lifecycle Manager also refers to most images by digest or uses images from registry.redhat.io where the image signatures can be verified.