How to manage Linux container registries

If we have a close look at LEGO® products, we can see that they are all made of the same building blocks. However, the composition of these blocks is the key differentiator for whether we are building a castle or space ship. It's pretty much the same for Podman, and its sibling projects Buildah, Skopeo, and CRI-O. However, instead of recycled plastic, the building blocks for our container tools are made of open source code. Sharing these building blocks allows us to provide rock-solid, enterprise-grade container tools. Features ship faster, bugs are fixed quicker, and the code is battle-tested. And, well, instead of bringing joy into playrooms, the container tools bring joy into data centers and workstations.

The most basic building block for our container tools is the containers/storage library, which locally stores and manages containers and container images. Going one level higher, we find the containers/image library. As the name suggests, this library deals with container images and is incredibly powerful. It allows us to pull and push images, manipulate images (e.g., change layer compression), inspect images along with their configuration and layers, and copy images between so-called image transports. A transport can refer to a local container storage, a container registry, a tar archive, and much more. Dan Walsh wrote a great blog post on the various transports that I highly recommend reading.

The image library also supports several configuration files where, without a doubt, the registries.conf is the most important one for the vast majority of users. I want to dedicate this blog post to the registries.conf configuration file, explain its various options and knobs, and how we can use them in production.

Managing container registries with registries.conf

The registries.conf configuration is in play whenever we push or pull an image. Or, more generally speaking, whenever we contact a container registry. That's an easy rule of thumb. The systemwide location is /etc/containers/registries.conf, but if you want to change that for a single user, you can create a new file at $HOME/.config/containers/registries.conf.

So let's dive right into it. In the following sections, we will go through some examples that explain the various configuration options in the registries.conf. The examples are real-world scenarios and may be a source of inspiration for tackling your individual use case.

Pulling by short names

Humans are lazy, and I am no exception to that. It is much more convenient to do a podman pull ubi8 rather than podman pull registry.access.redhat.com/ubi8:latest. I keep forgetting which image lives on which registry, and there are many images and a lot of registries out there. There is Docker Hub and Quay.io, plus registries for Amazon, Microsoft, Google, Red Hat, and many other Linux distributions.

Docker addressed our laziness by always resolving to the Docker Hub. A docker pull alpine will resolve to docker.io/library/alpine:latest, and docker pull repo/image:tag will resolve to docker.io/repo/image:tag (notice the specified repo). Podman and its sibling projects did not want to lock users into using one registry only, so short names can resolve to more than docker.io, and as you may expect, we can configure that in the registries.conf as follows:

unqualified-search-registries = ['registry.fedoraproject.org', 'registry.access.redhat.com', 'registry.centos.org', 'docker.io']

The above snippet is taken directly from the registries.conf in Fedora 33. It's a list of registries that are contacted in the specified order when pulling a short name image. If the image cannot be found on the first registry, Podman will attempt to pull from the second registry and so on. Buildah and CRI-O follow the same logic but note that Skopeo always normalizes to docker.io.

[ You might also like: What's the next Linux workload that you plan to containerize? ]

Searching images

Similar to the previous section on pulling, images are commonly searched by name. When doing a podman search, I usually do not know or simply forgot on which registry the given image lives. When using Docker, you can only search on the Docker Hub. Podman gives more freedom to users and allows for searching images on any registry. And unsurprisingly, registries.conf has a solution.

Similar to pulling, the unqualified-search-registries are also used when using a short name with podman search. A podman search foo will look for images named foo in all unqualified-search registries.

Large corporations usually have on-premises container registries. Integrating such registries in your workflow is as simple as adding them to the list of unqualified-search registries.

Short-name aliases

Newer versions of Podman, Buildah, and CRI-O ship with a new way of resolving short names, primarily by using aliases. Aliases are a simple TOML table [aliases] in the form "name" = "value", similar to how Bash aliases work. We maintain a central list of aliases together with the community upstream at github.com/containers/shortnames. If you own an image and want to have an alias, feel free to open a pull request or reach out to us.

Some distributions, like RHEL8, plan on shipping their own lists of short-names to help users and prevent them from accidentally pulling images from the wrong registry.

Explaining how short-name aliases work in detail would expand this blog post significantly, so if you are interested, please refer to an earlier blog post on short-name aliases.

Configuring a local container registry

Running a local container registry is quite common. I have one running all the time, so I can cache images and develop and test new features such as auto-updates in Podman. The bandwidth in my home office is limited, so I appreciate the fast pushes and pulls. Since everything is running locally, I don't need to worry about setting up TLS for the registry. That implies connecting to the registry via HTTP rather than via HTTPS. Podman allows you to do that by specifying --tls-verify=false on the command line, which will skip TLS verification and allow insecure connections.

An alternative approach to skipping TLS verification via the command line is by using the registries.conf. This may be more convenient, especially for automated scripts where we don't want to manually add command-line flags. Let's have a look at the config snippet below.

[[registry]]
location="localhost:5000"
insecure=true

The format of the registries.conf is TOML. The double braces of [[registry]] indicate that we can specify a list (or table) of [registry] objects. In this example, there is only one registry where the location (i.e., its address) is set to localhost:5000. That is where a local registry is commonly running. Whenever the containers/image library connects to a container registry with that location, it will look up its configuration and act accordingly. In this case, the registry is configured to be insecure, and TLS verification will be skipped. Podman and the other container tools can now talk to the local registry without getting the connections rejected.

Blocking a registry, namespace, or image

In case you want to prevent users or tools from pulling from a specific registry, you can do as follows.

[[registry]]
location="registry.example.org"
blocked=true

The blocked=true prevents connections to this registry, or at least to blocks pulling data from it.

However, it's surprisingly common among users to block only specific namespaces or individual images but not the entire registry. Let's assume that we want to stop users from pulling images from the namespace registry.example.org/namespace. The registries.conf will now look like this:

[[registry]]]
location="registry.example.org"
prefix="registry.example.org/example"
blocked=true

I just introduced a new config knob: prefix. A prefix instructs only to select the specified configuration when we attempt to pull an image that is matched by the specific prefix. For example, if we would run a podman pull registry.example.org/example/image:latest, the specified prefix would match, and Podman would be blocked from pulling the image. If you want to block a specific image, you can set it using the following:

prefix="registry.example.org/namespace/image"

Using a prefix is a very powerful tool to meet all kinds of use cases. It can be combined with all knobs of a [registry]. Note that using a prefix is optional. If none is specified, the prefix will default to the (mandatory) location.

Mirroring registries

Let's assume that we are running our workload in an air-gapped environment. All our servers are disconnected from the internet. There are many reasons for that. We may be running on the edge or running in a highly security-sensitive environment that forbids us from connecting to the internet. In this case, we cannot connect to the original registry but need to run a registry that mirrors the local network's contents.

A registry mirror is a registry that will be contacted before attempting to pull from the original one. It's a common use case and one of the oldest feature requests in the container ecosystem.

[[registry]]
location="registry.access.redhat.com"
[[registry.mirror]]
location="internal.registry.mirror"

With this configuration, when pulling the Universal Base Image via podman pull ubi8, the image would be pulled from the mirror instead of Red Hat's container registry.

Note that we can specify multiple mirrors that will be contacted in the specified order. Let's have a quick look at what that means:

[[registry]]
location="registry.example.com"
[[registry.mirror]]
location="mirror-1.com"
[[registry.mirror]]
location="mirror-2.com"
[[registry.mirror]]
location="mirror-3.com"

Let's assume we are attempting to pull the image registry.example.com/myimage:latest. Mirrors are contacted in the specified order (i.e., top-down), which means that Podman would first try to pull the image from mirror-1.com. If the image is not present or the pull fails for other reasons, Podman would contact mirror-2.com and so forth. If all mirror pulls fail, Podman will contact the main registry.example.com.

Note that mirrors also support the insecure knob. If you want to skip TLS verification for a specific mirror, just add insecure=true.

Remapping references

As we explored above, a prefix is used to select a specific [registry] in the registries.conf. While prefixes are a powerful means to block specific namespaces or certain images from being pulled, they can also be used to remap entire images. Similar to mirrors, we can use a prefix to pull from a different registry and a different namespace.

To illustrate what I mean by remapping, let's consider that we run in an air-gapped environment. We cannot access container registries since we are disconnected from the internet. Our workload is using images from Quay.io, Docker Hub, and Red Hat's container registry. While we could have one network-local mirror per registry, we could also just use one with the following config.

[[registry]]
prefix="quay.io"
location="internal.registry.mirror/quay"

[[registry]]
prefix="docker.io"
location="internal.registry.mirror/docker"

[[registry]]
prefix="registry.access.redhat.com"
location="internal.registry.mirror/redhat"

A podman pull quay.io/buildah/stable:latest will now instead pull internal.registry.mirror/quay/buildah/stable:latest. However, the pulled image will remain quay.io/buildah/stable:latest since the remapping and mirroring happen transparently to Podman and the other container tools.

As we can see in the snippet above, internal.registry.mirror is our network-local mirror that we are using to pull images on behalf of Quay.io, Docker Hub, and Red Hat's container registry. Images of each registry reside on separate namespaces on the registry (i.e., "quay", "docker", "redhat")—simple yet powerful trick to remap images when pulling. You may ask yourself how we can pre-populate the internal mirror with the images from the three registries. I do not recommend doing that manually but to use skopeo sync instead. With skopeo sync, a sysadmin can easily load all images onto a USB drive, bring that to an air-gapped cluster, and preload the mirror.

There are countless use cases where such remapping may help. For instance, when using another registry during tests, it may come in handy to transparently pull from another (testing or staging) registry than in production. No code changes are needed.

Tom Sweeney and Ed Santiago used the remapping to develop a creative solution to address the rate limits of Docker Hub. In late November 2020, Docker Hub started to limit the number of pulls per user in a given timeframe. At first, we were concerned because large parts of our testing systems, and continuous integration used Docker Hub images. But with a simple change to the registries.conf on our systems, Tom and Ed found a great solution. That spared us from the manual and tedious task of changing all images referring to docker.io in our tests.

Advanced configuration management via drop-on config files

Managing configurations is challenging. Our systems are updated all the time, and with the updates may come configuration changes. We may want to add new registries, configure new mirrors, correct previous settings or extend the default configuration of Fedora. There are many motivations, and for certain registries.conf supports it via so-called drop-in configuration files.

When loading the configuration, the containers/image library will first load the main configuration file at /etc/containers/registries.conf and then all files in the /etc/containers/registries.conf.d directory in alpha-numerical order.

Using such drop-in registries.conf files is straight forward. Just place a .conf file in the directory, and Podman will get the updated configuration. Note that tables in the config are merged while simple knobs are overridden. This means, in practice, that the [[registry]] table can easily be extended with new registries. If a registry with the same prefix already exists, the registry setting will be overridden. The same applies to the [aliases] table. Simple configuration knobs such as unqualified-search-registries are always overridden.

[ Getting started with containers? Check out this free course. Deploying containerized applications: A technical overview. ]

Conclusion

The registries.conf is a core building block of our container tools. It allows for configuring all kinds of properties when talking to a container registry. If you are interested in studying the configuration in greater detail, you can either do a man containers-registries.conf to the read the man page on your Linux machine or visit the upstream documentation.