How To Squash OpenShift Docker Build Layers To Remove Secrets

When you build a Docker image, a series of layers get cached to speed up subsequent builds. However, this caching can have adverse effects if an image layer contains secrets, such as credentials. A malicious person or process could find the layer containing your secrets and discover its contents.

Luckily, when performing Docker builds in OpenShift, you can quickly destroy secret layers by setting imageOptimizationPolicy to SkipLayers in your BuildConfig.

Let’s first consider the following Dockerfile:

1. FROM registry.redhat.io/ubi8/ubi:8.2

2. # Comes from an OCP secret also called ‘nexus-creds’

3. COPY nexus-creds nexus-creds

4. RUN curl -u $(cat nexus-creds/username):$(cat nexus-creds/password) -O http://example-nexusrepo-sonatype-nexus-demo.apps-crc.testing/repository/example-artifacts/jq && \

5. chown :0 jq && \

6. chmod 010 jq && \

7. mv jq /usr/local/bin && \

8. rm -rf nexus-creds

This Dockerfile installs jq to a UBI8-based image. However, rather than download jq from their releases page, it instead downloads from a local Nexus instance. It is common in the enterprise to download tools from a repository within the corporate network rather than from an upstream repository. However, to download from the corporate repository, you must first authenticate by providing your credentials. Credentials are provided in the example Dockerfile by copying them from a file in the build context (nexus-creds).

Here’s the million-dollar question: How do you remove these credentials from the image to not be revealed? If you think the answer is to remove the credentials in a RUN command (rm -rf nexus-creds), you are partially correct. That is a necessary step and will remove the credentials from the final container. However, it will not remove the credentials from the COPY layer. The COPY layer and the credentials it contains will still remain in the container registry. As a result, an attacker could steal your credentials by accessing this layer.

Below is partial output of the “oc start-build” command that builds the example Dockerfile. Here, you will see each of the layers pushed to the internal registry, one of them being the COPY layer, highlighted below.

1. Pushing image image-registry.openshift-image-registry.svc:5000/demo/example-image:latest ...

2. Getting image source signatures

3. Copying blob sha256:264aaa453271eaaf6bc92b521f62e38cbda4f01e1a3fbf50abdb8b466be58a64

4. Copying blob sha256:7f24bdb73d536337c7d82dee9c11e09acd3bacb0e1a76e98c6e82a8208e82284

5. Copying blob sha256:1c808c9ef8fef442a30a026190afef3c609840a0d53214d9054649394d26ba60

6. Copying blob sha256:1d1edefff77e9c0fd0518961b3e80aaffaab14e4b44b54b7a0b97cb0661cd39c

So, if removing the credentials in a RUN statement is only partially correct, what else can you do to prevent credentials from being stolen? Luckily, Red Hat addresses this by allowing you to provide a BuildConfig setting called imageOptimizationPolicy. When setting this to SkipLayers, the OpenShift build will squash each of the layers added to the base image (in other words, it will squash everything down to the FROM command). Since running “rm -rf nexus-creds” after the COPY command removes the credentials from the final layer, you can rest assured that the credentials will be removed entirely from both the image and the registry itself by squashing the layers with imageOptimizationPolicy.

Below is an example of a BuildConfig that uses the imageOptimizationPolicy: SkipLayers setting.

1. apiVersion: build.openshift.io/v1

2. kind: BuildConfig

3. metadata:

4. name: example-image

5. spec:

6. output:

7. to:

8. kind: ImageStream

9. name: example-image

10. source:

11. binary: {}

12. type: Binary

13. secrets:

14. - destinationDir: nexus-creds

15. secret:

16. name: nexus-creds

17. strategy:

18. dockerStrategy:

19. imageOptimizationPolicy: SkipLayers

20. pullSecret:

21. name: pull-secret

22. dockerfilePath: Dockerfile

23. type: Docker

When this BuildConfig is applied, the resulting image will have only three layers (two from the base image and one from the image we are building off that base). Below is the ending output of “oc start-build” after the imageOptimizationPolicy setting is applied:

1. Pushing image image-registry.openshift-image-registry.svc:5000/demo/example-image:latest ...

2. Getting image source signatures

3. Copying blob sha256:7f24bdb73d536337c7d82dee9c11e09acd3bacb0e1a76e98c6e82a8208e82284

4. Copying blob sha256:264aaa453271eaaf6bc92b521f62e38cbda4f01e1a3fbf50abdb8b466be58a64

5. Copying blob sha256:9e60edd0480d9cd1c51f682a2978582a42951172f507f57da0b18656f71df037

Only three layers this time instead of four. If you were to access your image registry or pull this image down from outside the cluster, there would be no trace of the COPY layer or your credentials for this given image and tag.

Thanks For Reading

Hopefully, this helps you prevent secrets from being exposed within intermediate layers stored in your image registry. For more information, be sure to check out Red Hat’s documentation on imageOptimizationPolicy and squashing layers in Docker builds.