Running Testcontainers in OpenShift Pipelines With Docker-in-Docker

Please note that the capabilities covered in this post are not officially supported by Red Hat. Container technologies have had a very strong influence in application development. In addition to the well-documented value of containers related to deployment and management of applications, one of the biggest impacts has been the ability to create sets of full integration tests that can run anywhere.

The traditional way to test applications was heavily rooted in mocking due to the inability of the developers to run external services in their local development environment. The reason for mocking is to be able to programmatically replicate expected results from an external service, such as a database or messaging broker. Developers would create mock implementations using framework like Mockito and substitute these mock implementations when running in an environment where the services are not available. The problems pop up due to the limitations associated with mocking such as the inability to properly test database schema changes or transactions.

With containers, these service availability constraints can be eliminated. Developers can now spin up all kinds of infrastructure and services quickly and easily. Developers can use tools like docker or podman to start the external services in whatever environment they like and use those services for testing. For example, instead of writing a bunch of mock implementations for data access, which require their own care and maintenance, developers can now run suites of tests that can cover the full stack, not just the application code.

Enter Testcontainers

The next step in the evolution was to bring container orchestration into the testing life cycle. The first iteration was based on the build tool life cycles. For example, in Maven, a developer could use a Maven plugin to create and destroy resources as needed in the Maven life cycle. However, these quickly evolved and were replaced by more agile Java libraries like Testcontainers. The Testcontainers library provided a lightweight, easily customizable set of APIs  to plug-in containers of all types directly into JUnit. At first glance, Testcontainers seems to be the ultimate enabler for building a very powerful and thorough set of testing capabilities for any project. Developers can install Docker on their local machine and run full integration tests using a wide array of industry standard services.

Testcontainers finds my local daemon managed by Docker Desktop

The Issue: CICD

The problem comes from Testcontainers native integration with Docker. Testcontainers expects a docker daemon to be generally available on the host in which it is running, typically located at /var/run/docker.sock. If you are running some legacy CICD products that run on a single host, this is generally not a problem. However, when you want to use something that is container based, such as Tekton, then the issue rears its ugly head. The containers used to perform builds do not expose a Docker daemon and therefore the Testcontainers library will not be able to do its job.

Where is my unix domain socket for Docker? It is not there.

Docker-in-Docker

For the solution, we need to be able to run Docker. However, because we are running a Tekton pipeline, we are going to be running in containers, so the solution is to be able to run Docker in a container. This is where Docker-in-Docker comes in handy. According to the Docker blog:

What’s special in my dind? Almost nothing! It is built with a regular Dockerfile. Let’s see what is in that Dockerfile.

First, it installs a few packages: lxc and iptables (because Docker needs them), and ca-certificates (because when communicating with the Docker index and registry, Docker needs to validate their SSL certificates).

The Dockerfile also indicates that /var/lib/docker should be a volume. This is important, because the filesystem of a container is an AUFS mountpoint, composed of multiple branches; and those branches have to be “normal” filesystems (i.e. not AUFS mountpoints). In other words, /var/lib/docker, the place where Docker stores its containers, cannot be an AUFS filesystem. Therefore, we instruct Docker that this path should be a volume. Volumes have many purposes, but in this scenario, we use them as a pass-through to the “normal” filesystem of the host machine. The /var/lib/docker directory of the nested Docker will live somewhere in /var/lib/docker/volumes on the host system.

Now that we have a solution for being able to run Docker inside of a container, we need to figure out how to plug this into the task.

Tekton Sidecar

Luckily, Tekton has an answer for this. Sidecars allow additional containers to be configured and spun up on a task pod. For our example, we would be looking for a Maven container to execute the appropriate goals, but we need to have a running Docker daemon located at /var/run/docker.sock and available to the JUnit test life cycle.

The Task

Here is the full yaml for an example Maven build task:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: maven-build-task
spec:
  workspaces:
    - name: source
    - name: maven-settings
  params:
    - name: MAVEN_IMAGE
      type: string
      description: Maven base image
      default: gcr.io/cloud-builders/mvn@sha256:57523fc43394d6d9d2414ee8d1c85ed7a13460cbb268c3cd16d28cfb3859e641 #tag: latest
    - name: GOALS
      description: maven goals to run
      type: array
      default:
        - "package"
    - name: MAVEN_MIRROR_URL
      description: The Maven repository mirror url
      type: string
      default: ""
    - name: SERVER_USER
      description: The username for the server
      type: string
      default: ""
    - name: SERVER_PASSWORD
      description: The password for the server
      type: string
      default: ""
    - name: PROXY_USER
      description: The username for the proxy server
      type: string
      default: ""
    - name: PROXY_PASSWORD
      description: The password for the proxy server
      type: string
      default: ""
    - name: PROXY_PORT
      description: Port number for the proxy server
      type: string
      default: ""
    - name: PROXY_HOST
      description: Proxy server Host
      type: string
      default: ""
    - name: PROXY_NON_PROXY_HOSTS
      description: Non proxy server host
      type: string
      default: ""
    - name: PROXY_PROTOCOL
      description: Protocol for the proxy ie http or https
      type: string
      default: "http"
    - name: CONTEXT_DIR
      type: string
      description: >-
        The context directory within the repository for sources on
        which we want to execute maven goals.
      default: "."
  steps:
    - name: mvn-settings
      image: registry.access.redhat.com/ubi8/ubi-minimal:8.2
      script: |
        #!/usr/bin/env bash
        [[ -f $(workspaces.maven-settings.path)/settings.xml ]] && \
        echo 'using existing $(workspaces.maven-settings.path)/settings.xml' && exit 0
        cat > $(workspaces.maven-settings.path)/settings.xml <<EOF
        <settings>
          <servers>
            <!-- The servers added here are generated from environment variables. Don't change. -->
            <!-- ### SERVER's USER INFO from ENV ### -->
          </servers>
          <mirrors>
            <!-- The mirrors added here are generated from environment variables. Don't change. -->
            <!-- ### mirrors from ENV ### -->
          </mirrors>
          <proxies>
            <!-- The proxies added here are generated from environment variables. Don't change. -->
            <!-- ### HTTP proxy from ENV ### -->
          </proxies>
        </settings>
        EOF
        xml=""
        if [ -n "$(params.PROXY_HOST)" -a -n "$(params.PROXY_PORT)" ]; then
          xml="<proxy>\
            <id>genproxy</id>\
            <active>true</active>\
            <protocol>$(params.PROXY_PROTOCOL)</protocol>\
            <host>$(params.PROXY_HOST)</host>\
            <port>$(params.PROXY_PORT)</port>"
          if [ -n "$(params.PROXY_USER)" -a -n "$(params.PROXY_PASSWORD)" ]; then
            xml="$xml\
                <username>$(params.PROXY_USER)</username>\
                <password>$(params.PROXY_PASSWORD)</password>"
          fi
          if [ -n "$(params.PROXY_NON_PROXY_HOSTS)" ]; then
            xml="$xml\
                <nonProxyHosts>$(params.PROXY_NON_PROXY_HOSTS)</nonProxyHosts>"
          fi
          xml="$xml\
              </proxy>"
          sed -i "s|<!-- ### HTTP proxy from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
        fi
        if [ -n "$(params.SERVER_USER)" -a -n "$(params.SERVER_PASSWORD)" ]; then
          xml="<server>\
            <id>serverid</id>"
          xml="$xml\
                <username>$(params.SERVER_USER)</username>\
                <password>$(params.SERVER_PASSWORD)</password>"
          xml="$xml\
              </server>"
          sed -i "s|<!-- ### SERVER's USER INFO from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
        fi
        if [ -n "$(params.MAVEN_MIRROR_URL)" ]; then
          xml="    <mirror>\
            <id>mirror.default</id>\
            <url>$(params.MAVEN_MIRROR_URL)</url>\
            <mirrorOf>central</mirrorOf>\
          </mirror>"
          sed -i "s|<!-- ### mirrors from ENV ### -->|$xml|" $(workspaces.maven-settings.path)/settings.xml
        fi
    - name: mvn-goals
      image: $(params.MAVEN_IMAGE)
      workingDir: $(workspaces.source.path)/$(params.CONTEXT_DIR)
      command: ["/usr/bin/mvn"]
      args:
        - -s
        - $(workspaces.maven-settings.path)/settings.xml
        - "$(params.GOALS)"
        - -ntp
      volumeMounts:
        - mountPath: /var/run/
          name: dind-socket
  sidecars:
    - image: docker:20.10-dind
      name: docker
      securityContext:
        privileged: true
      volumeMounts:
        - mountPath: /var/lib/docker
          name: dind-storage
        - mountPath: /var/run/
          name: dind-socket
  volumes:
    - name: dind-storage
      emptyDir: {}
    - name: dind-socket
      emptyDir: {}

In this task, the Docker sidecar is started, and the Docker daemon is mounted on a shared volume, dind-socket. That same volume is mounted in the Maven container on the mount path expected by Testcontainers, namely /var/run. This gives us access to the running Docker daemon and makes it available for use by Testcontainers.

Adding in the Proper Permissions

If you try to run this task in OpenShift without any additional configuration, it will not work. OpenShift, by default, does not allow containers to be run in privileged mode which enhances the overall platform security. But in this case, we need to run in privileged mode because of Docker. Therefore, we need to update the security for the service account running our tasks.

We want to isolate the permissions to a single privileged service account to the CICD  namespace. Of course, this is assuming all of your pipelines run in a single namespace and are generally managed as a centralized service. If this is not the case, then each pipeline administrator will need to set up their own privileged service account for their own namespace. To accomplish this, let’s create a new service account and provide the correct SCC  to it:

oc project cicd
oc create sa dind
oc adm policy add-scc-to-user privileged -z dind

This serviceAccountName can be specified in the TriggerTemplate for the pipeline and can be narrowed to only apply to the Maven build step that needs the access:

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
  name: maven-build-trigger-template
spec:
  params:
    - name: git-repo-url
      description: The git repository url
    - name: git-repo-name
      description: The name of the rep
    - name: git-revision
      description: The git revision
    - name: git-ref
      description: The name of the ref

  resourcetemplates:
  - apiVersion: tekton.dev/v1beta1
    kind: PipelineRun
    metadata:
      generateName: maven-build-pipeline-$(tt.params.git-repo-name)-
    spec:
      serviceAccountName: pipeline
      serviceAccountNames:
        - taskName: maven-build-task
           serviceAccountName: dind
      pipelineRef:
        name: maven-build-pipeline
      params:
        - name: git-repo-url
          value: $(tt.params.git-repo-url)
        - name: git-repo-name
          value: $(tt.params.git-repo-name)
        - name: git-revision
          value: $(tt.params.git-revision)
        - name: git-ref
          value: $(tt.params.git-ref)
      workspaces:
      - name: workspace
        volumeClaimTemplate:
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 500Mi

Happy building!