Container Image Signing

Red Hat engineers have been working to more securely distribute container images. In this post we look at where we’ve come from, where we need to go, and how we hope to get there.

History

When the Docker image specification was introduced it did not have a cryptographic verification model. The most significant reason (for not having one) was the lack of a reliable checksum hash of image content. Two otherwise identical images could have different checksum values. Without a consistent tarsum mechanism, cryptographic verification would be very challenging. With Docker version 1.10, checksums are more consistent and could be used as a stable reference for

cryptographic verification. The version 2 image format provides an image manifest digest hash value that is useful for this.

New Packaging Format, Old Problem

Language-specific package formats such as npm and pip have struggled to implement signing models. The Docker image format is unique but ultimately is just another software packaging format. Meanwhile, software vendors and system administrators have had years of experience with securing RPM packages. While RPM signatures have served the industry for a long time, there are improvements that can be made to satisfy the following use cases:

• Verify image integrity and authorship offline or from within an “air gapped” data center
• Support multiple signatures to assert chain of trust/custody
• Easily integrate with existing enterprise infrastructure

The Human Factor

One of the guiding principles that should be used when designing a security model is to recognize the importance of how humans interact with technology. If the interaction is awkward or cumbersome it leads to insecurity. Software vendors, developers, and system administrators should be able to understand the security model with minimal documentation.

We want a system that serves developers and system administrators and doesn’t get in the way. I’m looking at you, “self-signed certificate” and “proceed insecure?” warnings. We don’t want to force application developers into a model that requires centrally managed CAs or container-specific signing servers. Complex security models often result in insecure systems, whether through a false sense of security (XCodeGhost, anyone?) or user workarounds that circumvent security.

We want a system that passes rigorous security review, namely it must:

1. Use well-established protocols, standards, and processes.
2. Be robust, but (also) simple enough to be understood by security auditors.
3. Easily integrate with existing infrastructure (including existing signing servers and processes).

Introducing: Simple Signing

The approach Red Hat engineers have developed is powerfully simple. Each signature generates a signature claim, in the form of a simple json file, one that includes the image manifest digest hash.  The following illustrates how an example signature claim might look in practice:

{
    "critical": {
           "identity": {
               "docker-reference": "testing/manifest"
           },
           "image": {
               "Docker-manifest-digest": "sha256:20be...fe55"
           },
           "type": "atomic container signature"
    },
    "optional": {
           "creator": "atomic",
           "timestamp": 1458239713
    }
}

Signing involves using the OpenPGP standard to encrypt the signature claim text file with a private key, creating a binary signature file.

Sign: generate a signature claim

While this is being integrated into OpenShift and Atomic Registry as a first-class object, this file can be proliferated on systems all over the world and may be served independent of the image and originating registry.

To verify an image, the signature claim is decoded using the signer’s public key. If the image manifest digest hash matches you have a very high level of certainty the image from the signer has been unaltered.

Verify: signature claim must match image manifest digest

The signature claim files are simple to distribute using a basic web server that relies on standard transport security protocols.

Chain of Trust and Policy

With this model any number of signature claims may be created. Different parties may assert trust against the same image hash. This provides an opportunity for software vendors and enterprise users to add policy around which signatures are required to run on a given cluster. For example, policy could require three signatures to run on the production cluster: from Red Hat, a certified software vendor and the corporate key, or even multiple specialist teams within a company such as operations, security, middleware, and individual developer teams.

Integrate All the Things

Finally, we want to enable integration. This model will work with your existing keys and workflows for securing keys, such as signing servers, hardware security modules, or even your home-grown script for signing with a key on an USB flash drive. There’s no need to retool for a container-specific key format and key management tools. We’re simply trying to sign container images, not design a new key format with specialized key management tooling.

What’s Next?

Red Hat is contributing this work as part of an open source shared library that can be used by anyone to work towards common signature semantics across technologies. It also provides a policy framework where trusted public keys and other aspects only need to be defined once per system. Red Hat is driving signing innovation in an inclusive, open, and community driven approach to make containers more secure for us all.

Interested in joining this work? Thoughts?  Questions?  We’d love to know what you think.  Reach out using the comments section (below).