Speeding up container image builds with Buildah

A few months ago, I wrote an article on speeding up container builds inside of a container. That article concentrated on the speed of pulling container images, and different ways to prepopulate the image store, using volume mounts from the host and the Buildah concept of "additional stores." 

Buildah is a command-line tool for building Open Container Initiative-compatible (that means Docker and Kubernetes-compatible, too) images quickly and easily. Buildah is easy to incorporate into scripts and build pipelines, and best of all, it doesn't require a running container daemon to build its image.

This article will address a second problem with build speed when using dnf/yum commands inside containers. Note that in this article I will use the name dnf (which is the upstream name) instead of what some downstreams use (yum) These comments apply to both dnf and yum.

Download speed

Did you ever notice that sometimes when you run dnf -y update or dnf -y install for the first time in a while, the command pauses for a long time before it even starts pulling down RPMs? What is going on?

The first thing that dnf does is download huge cache files. These files are written in XML and contain every single package in the remote repository, including lots of data about the package. They even contain every path within the package. This data is needed so that when you can run something like dnf -y install /usr/bin/httpd and then dnf figures out the package to be installed. Many packages contain commands like (requires: /usr/bin/sendmail) that take advantage of this feature, allowing dnf to pull an appropriate package to satisfy the need.

Pulling these huge files—and more importantly, processing these files—can take over a minute. This data is used by libsolv, so it must be converted to the solv format, which is slow. Speed is not a huge issue when you only do this periodically on your host, but when it comes to building containers, this is a much bigger deal.

Dockerfile syntax

Although Buildah allows you to build container images directly in the shell, most people use Dockerfiles and Containerfiles to build containers and define reproducible image recipes. Buildah defaults to look for a Containerfile and a Dockerfile now. Each share the same syntax, so I will use Containerfile for the rest of this document.

A common thing to do in a Containerfile is to use a syntax like:

FROM ubi8  
RUN dnf -y update; dnf -y install nginx; dnf -y clean all  
…  
RUN dnf -y install jboss; dnf -y clean all  

Let’s examine the dnf lines. The first dnf line:

(dnf -y update; dnf -y install nginx; dnf -y clean all):

1. Updates all packages within the container.
2. Installs the selected package nginx.
3. Cleans all.

Since the ubi8 image was probably created a while ago and its /var/cache/dnf directory probably does not exist inside the container image, dnf must pull down the XML cache file and process it. Then, dnf installs the actual packages before dnf -y clean all removes all of the excess data that the earlier commands placed into the image like log and cache files.

Users are recommended to run clean all to keep the image as small as possible. Each RUN command creates a new layer, and even if you remove the content in a later RUN command, the initial layer will contain all of the content. This means that everyone that ever pulls your image will end up pulling the logs and cache files. Now, if your Containerfile contains one or more dnf commands, you will pay the price over and over again. Not only that but every time you rebuild this image, you will pay that price yet again. If you are on a build server, every container image that you build will download these XML files over and over, wasting tons of resources and time.

Buildah with overlay mounts

We saw the problem described above and thought we could handle this in a better way. Couldn’t we just pull the XML data to the host, process it on the host, and volume mount it into the containers? Perhaps we could set up a cron job or a systemd timer that does a dnf makecache once for each version of OSs that you will build container images for? You could run this job once or multiple times a day on the host and then have all of the container builders volume mount the appropriate caches into the buildah containers.

Well, Buildah supports volume mounting directories from the host into containers. That should solve the problem, and it does. BUT, the containers often want to write to this directory, if they have to update the cache, so the cache needs to be mounted into the containers read/write. This causes a huge security hole. Imagine a situation in which a hostile container build wrote content to this cache that a later container builder read. It could possibly trick the second container into installing hacked software. We need a solution where the content is mounted into the container, can not be written by the container, yet still be written to from the container’s perspective. This is fundamentally what an Overlay mount point is.

Overlay file system mounts up a lower directory and then attaches an upper directory to the merged mount point. When a process writes to a new file to the merged directory, the new file gets written to the upper directory. When a process modifies an existing file in the lower directory, the kernel copies-up the file from the lower directory to the upper directory and allows the process to modify the file in the upper directory.

We introduced the concept of the Overlay mount to Buildah. Now you can run builds with

buildah bud -v /var/cache/dnf:/var/cache/dnf:O -f /tmp/Containerfile /tmp  

Dnf inside of the container will still check to see if there is newer content at the repos, and will pull the content down if it exists. But if the content on the host is up to date, it will quickly use the host's cache. I would recommend that you update the hosts cache at least once per day.

One additional feature we added for the Buildah Overlay mount is to destroy the upper directory on every RUN directive. Recall in our example, we used multiple RUN commands that were each executing a dnf -y clean all. The dnf -y clean all command causes the upper directory to show all of the content from the lower as deleted. If the next dnf command shared the previous upper it would see the cache as empty and would have to pull down the XML datastore and process it. Removing the upper directory means that each dnf command will again see the lower directory from the host, and continue to share the hosts cache.

Speed difference

I am going to create a simple Containerfile containing two dnf run commands.

FROM fedora:31  
RUN dnf -y install net-utils; dnf -y clean all  
RUN dnf -y install iputils; dnf -y clean all  

Running this locally on my Fedora 31 box

# time -f "Elapsed Time: %E" buildah bud -f Containerfile .  
STEP 1: FROM fedora:31  
STEP 2: RUN dnf -y install procps-ng; dnf -y clean all  
Fedora Modular 31 - x86_64 2.0 MB/s | 5.2 MB 00:02  
Fedora Modular 31 - x86_64 - Updates 1.6 MB/s | 4.0 MB 00:02  
Fedora 31 - x86_64 - Updates 4.2 MB/s | 19 MB 00:04  
Fedora 31 - x86_64 1.8 MB/s | 71 MB 00:39  
Last metadata expiration check: 0:00:01 ago on Wed Feb 5 13:55:54 2020.  
Dependencies resolved.  
================================================================================  
Package Architecture Version Repository Size  
================================================================================  
Installing:  
procps-ng x86_64 3.3.15-6.fc31 fedora 326 k

Transaction Summary  
================================================================================  
Install 1 Package

Total download size: 326 k  
Installed size: 966 k  
Downloading Packages:  
procps-ng-3.3.15-6.fc31.x86_64.rpm 375 kB/s | 326 kB 00:00  
--------------------------------------------------------------------------------  
Total 218 kB/s | 326 kB 00:01  
Running transaction check  
Transaction check succeeded.  
Running transaction test  
Transaction test succeeded.  
Running transaction  
Preparing : 1/1  
Installing : procps-ng-3.3.15-6.fc31.x86_64 1/1  
Running scriptlet: procps-ng-3.3.15-6.fc31.x86_64 1/1  
Verifying : procps-ng-3.3.15-6.fc31.x86_64 1/1

Installed:  
procps-ng-3.3.15-6.fc31.x86_64

Complete!  
33 files removed  
STEP 3: RUN dnf -y install iputils; dnf -y clean all  
Fedora Modular 31 - x86_64 741 kB/s | 5.2 MB 00:07  
Fedora Modular 31 - x86_64 - Updates 928 kB/s | 4.0 MB 00:04  
Fedora 31 - x86_64 - Updates 3.8 MB/s | 19 MB 00:05  
Fedora 31 - x86_64 7.9 MB/s | 71 MB 00:08  
Last metadata expiration check: 0:00:01 ago on Wed Feb 5 13:57:13 2020.  
Dependencies resolved.  
================================================================================  
Package Architecture Version Repository Size  
================================================================================  
Installing:  
iputils x86_64 20190515-3.fc31 fedora 141 k

Transaction Summary  
================================================================================  
Install 1 Package

Total download size: 141 k  
Installed size: 387 k  
Downloading Packages:  
iputils-20190515-3.fc31.x86_64.rpm 252 kB/s | 141 kB 00:00  
--------------------------------------------------------------------------------  
Total 141 kB/s | 141 kB 00:01  
Running transaction check  
Transaction check succeeded.  
Running transaction test  
Transaction test succeeded.  
Running transaction  
Preparing : 1/1  
Installing : iputils-20190515-3.fc31.x86_64 1/1  
Running scriptlet: iputils-20190515-3.fc31.x86_64 1/1  
Verifying : iputils-20190515-3.fc31.x86_64 1/1

Installed:  
iputils-20190515-3.fc31.x86_64

Complete!  
33 files removed  
STEP 4: COMMIT  
Getting image source signatures  
Copying blob ac0b803c5612 skipped: already exists  
Copying blob 922380d685bc done  
Copying config 566e2afbb4 done  
Writing manifest to image destination  
Storing signatures  
566e2afbb417f0119109578a87950250b566a3b4908868627975a4c7428accfb  
566e2afbb417f0119109578a87950250b566a3b4908868627975a4c7428accfb

Elapsed Time: 2:15.00  

This run took 2 minutes and 15 seconds to create a new container image with the two new packages.

Now let’s try this with an Overlay mount from the host.

# dnf -y makecache  
# time -f "Elapsed Time: %E" buildah bud -v /var/cache/dnf:/var/cache/dnf:O -f Containerfile .  
STEP 1: FROM fedora:31  
STEP 2: RUN dnf -y install procps-ng; dnf -y clean all  
Last metadata expiration check: 0:02:34 ago on Wed Feb 5 13:51:54 2020.  
Dependencies resolved.  
================================================================================  
Package Architecture Version Repository Size  
================================================================================  
Installing:  
procps-ng x86_64 3.3.15-6.fc31 fedora 326 k

Transaction Summary  
================================================================================  
Install 1 Package

Total download size: 326 k  
Installed size: 966 k  
Downloading Packages:  
procps-ng-3.3.15-6.fc31.x86_64.rpm 496 kB/s | 326 kB 00:00  
--------------------------------------------------------------------------------  
Total 245 kB/s | 326 kB 00:01  
Running transaction check  
Transaction check succeeded.  
Running transaction test  
Transaction test succeeded.  
Running transaction  
Preparing : 1/1  
Installing : procps-ng-3.3.15-6.fc31.x86_64 1/1  
Running scriptlet: procps-ng-3.3.15-6.fc31.x86_64 1/1  
Verifying : procps-ng-3.3.15-6.fc31.x86_64 1/1

Installed:  
procps-ng-3.3.15-6.fc31.x86_64

Complete!  
285 files removed  
STEP 3: RUN dnf -y install iputils; dnf -y clean all  
Last metadata expiration check: 0:02:41 ago on Wed Feb 5 13:51:54 2020.  
Dependencies resolved.  
================================================================================  
Package Architecture Version Repository Size  
================================================================================  
Installing:  
iputils x86_64 20190515-3.fc31 fedora 141 k

Transaction Summary  
================================================================================  
Install 1 Package

Total download size: 141 k  
Installed size: 387 k  
Downloading Packages:  
iputils-20190515-3.fc31.x86_64.rpm 556 kB/s | 141 kB 00:00  
--------------------------------------------------------------------------------  
Total 222 kB/s | 141 kB 00:00  
Running transaction check  
Transaction check succeeded.  
Running transaction test  
Transaction test succeeded.  
Running transaction  
Preparing : 1/1  
Installing : iputils-20190515-3.fc31.x86_64 1/1  
Running scriptlet: iputils-20190515-3.fc31.x86_64 1/1  
Verifying : iputils-20190515-3.fc31.x86_64 1/1

Installed:  
iputils-20190515-3.fc31.x86_64

Complete!  
285 files removed  
STEP 4: COMMIT  
Getting image source signatures  
Copying blob ac0b803c5612 skipped: already exists  
Copying blob 524bb3b83d61 done  
Copying config 0f82aa6064 done  
Writing manifest to image destination  
Storing signatures  
0f82aa6064814ff3dcb603c34c75e516e00817811681b83b8632f3e9b694e518  
0f82aa6064814ff3dcb603c34c75e516e00817811681b83b8632f3e9b694e518  
Elapsed Time: 0.17.44  

With the Overlay mount, we were able to build a new image with the two additional packages in 17 seconds instead of 2 minutes and 15 seconds. That is almost 8 times faster to build the same container image.

Now this shows that if you build images on a host operating system that has the dnf metadata pre-cached you can accelerate the install speed by a HUGE amount. But what if your build system builds images for other versions of the OS? Say you want to build images for Fedora 30 as well as Fedora 31. Note: this would also work on a RHEL8 system, where you might want to build RHEL7 and perhaps even RHEL6 images.
Dnf includes a cool feature where you can specify different releases when pulling content, using the --releasever option. Dnf also allows you to specify alternative directories to place the cachedir, --setopt=cachedir.

In the following example, I am going to pull down two caches on the host and then use Buildah in command line mode.

# dnf -y makecache --releasever=31 --setopt=cachedir=/var/cache/dnf/31  
# dnf -y makecache --releasever=30 --setopt=cachedir=/var/cache/dnf/30  
# ctr31=$(buildah from fedora:31)  
# time -f 'Elapsed Time: %E' buildah run -v /var/cache/dnf/31:/var/cache/dnf:O ${ctr31} dnf -y install iputils  
Last metadata expiration check: 0:00:15 ago on Wed Feb 5 14:17:41 2020.  
Dependencies resolved.  
================================================================================  
Package Architecture Version Repository Size  
================================================================================  
Installing:  
iputils x86_64 20190515-3.fc31 fedora 141 k

Transaction Summary  
================================================================================  
Install 1 Package

Total download size: 141 k  
Installed size: 387 k  
Downloading Packages:  
iputils-20190515-3.fc31.x86_64.rpm 192 kB/s | 141 kB 00:00  
--------------------------------------------------------------------------------  
Total 107 kB/s | 141 kB 00:01  
Running transaction check  
Transaction check succeeded.  
Running transaction test  
Transaction test succeeded.  
Running transaction  
Preparing : 1/1  
Installing : iputils-20190515-3.fc31.x86_64 1/1  
Running scriptlet: iputils-20190515-3.fc31.x86_64 1/1  
Verifying : iputils-20190515-3.fc31.x86_64 1/1

Installed:  
iputils-20190515-3.fc31.x86_64

Complete!  
Elapsed Time: 0:06.85

# ctr30=$(buildah from fedora:30)  
# time -f 'Elapsed Time: %E' buildah run -v /var/cache/dnf/30:/var/cache/dnf:O ${ctr30} dnf -y install iputils  
Last metadata expiration check: 0:00:15 ago on Wed Feb 5 14:17:47 2020.  
Dependencies resolved.  
================================================================================  
Package Architecture Version Repository Size  
================================================================================  
Installing:  
iputils x86_64 20180629-4.fc30 fedora 123 k

Transaction Summary  
================================================================================  
Install 1 Package

Total download size: 123 k  
Installed size: 351 k  
Downloading Packages:  
iputils-20180629-4.fc30.x86_64.rpm 370 kB/s | 123 kB 00:00  
--------------------------------------------------------------------------------  
Total 138 kB/s | 123 kB 00:00  
Running transaction check  
Transaction check succeeded.  
Running transaction test  
Transaction test succeeded.  
Running transaction  
Preparing : 1/1  
Installing : iputils-20180629-4.fc30.x86_64 1/1  
Running scriptlet: iputils-20180629-4.fc30.x86_64 1/1  
Verifying : iputils-20180629-4.fc30.x86_64 1/1

Installed:  
iputils-20180629-4.fc30.x86_64

Complete!  
Elapsed Time: 0:08.88  

As you can see, we were able to run Buildah containers using the dnf cache from two different releases of Fedora from the same build host and the containers for Fedora 31 took 6+ seconds and the Fedora 30 build took 8+ seconds.

Note: I chose a subdirectory of /var/cache/dnf for the cache files, to make sure SELinux labels were correct. Just running dnf clean all will not clean /var/cache/dnf/31. You would need to execute dnf clean all --setopt=cachedir=/var/cache/dnf/31 to properly clean repo cached files, but some artifacts will remain anyway (gpg keys, empty directories).

Now just to see how long this would take to run the build on Fedora 31 without the Overlay mount.

# ctr31=$(buildah from fedora:31)  
# time -f 'Elapsed Time: %E' buildah run ${ctr31} dnf -y install iputils  
Fedora Modular 31 - x86_64 1.2 MB/s | 5.2 MB 00:04  
Fedora Modular 31 - x86_64 - Updates 875 kB/s | 4.0 MB 00:04  
Fedora 31 - x86_64 - Updates 2.4 MB/s | 19 MB 00:07  
Fedora 31 - x86_64 1.7 MB/s | 71 MB 00:41  
Dependencies resolved.  
================================================================================  
Package Architecture Version Repository Size  
================================================================================  
Installing:  
iputils x86_64 20190515-3.fc31 fedora 141 k

Transaction Summary  
================================================================================  
Install 1 Package

Total download size: 141 k  
Installed size: 387 k  
Downloading Packages:  
iputils-20190515-3.fc31.x86_64.rpm 279 kB/s | 141 kB 00:00  
--------------------------------------------------------------------------------  
Total 129 kB/s | 141 kB 00:01  
Running transaction check  
Transaction check succeeded.  
Running transaction test  
Transaction test succeeded.  
Running transaction  
Preparing : 1/1  
Installing : iputils-20190515-3.fc31.x86_64 1/1  
Running scriptlet: iputils-20190515-3.fc31.x86_64 1/1  
Verifying : iputils-20190515-3.fc31.x86_64 1/1

Installed:  
iputils-20190515-3.fc31.x86_64

Complete!  
Elapsed Time: 1:29.85  

In this case, it took almost 1.5 minutes to run the same container. Buildah with Overlay mounts ran 14 times faster.

Rootless Containers

All my examples up until now have been running the builds using Containerfiles as root. But you can do this with rootless containers as well. In the next couple of examples, I will have Buildah run containers using the buildah run syntax to demonstrate using the cache.

Executing

$ buildah run -v /var/cache/dnf/30:/var/cache/dnf:O ${ctr30} dnf -y install iputils  

works fine. As long as the user can read the /var/cache/dnf/30 directory, the lower directory can be read. But you have to rely on something on the host to update the cache periodically.

If users want they can even use dnf to create the cache in their home directory.

$ dnf -y makecache --releasever=30 --setopt=cachedir=$HOME/dnfcache  
$ chcon --reference /var/cache/dnf -R $HOME/dnfcache  
$ ctr30=$(buildah from fedora:30)  
$ buildah run -v $HOME/dnfcache:/var/cache/dnf:O ${ctr30} dnf -y install iputils  

Notice I had to change the SELinux label of the $HOME/dnfcache directory so that SELinux would allow the containers to read the lower directory for the Overlay mount.

Conclusion

Speeding up the build of containers requires an understanding of what is going on when you are installing packages. Pre-caching dnf data on the host and using Overlay mounts to mount the cache into the container with Buildah can greatly increase the speed of builds and reduce the number of resources required to support a build farm.

Buildah equals simplicity but it also has some great features like Overlay mounts and additional stores which can help you speed up your container image builds.

[ New to containers? Download the Containers Primer and learn the basics of Linux containers. ]