Be careful when pulling images by short name

Many people pull container images from public registries, full of user generated content. Do they  really know who built them, how they were built, or whether they are trustworthy? There are several reports about compromised images which trick people into downloading them. Today many users understand that they should always pull a trusted container image from a trusted source, for example pulling the Red Hat Universal Base Image from the Red Hat Ecosystem Catalog.

Sadly, you can still be compromised if you don’t use the full URL of the image.

Users pulling without specifying the full path, commonly referred to as a short name, leaves them open to a particularly nasty form of attack, called image squatting. 

The ramifications of using short names to pull container images

When pulling container images, many veteran users refer to an image in a repository without using its fully qualified image registry path. For example, instead of using:

$ podman pull registry.access.redhat.com/rhe7/etcd

They use the following and assume it will pull the right image:

​$ podman pull rhel7/etcd
Trying to pull registry.access.redhat.com/rhel7/etcd…
Getting image source signatures
…

Another common scenario is with Dockerfiles. Short names are often used in the FROM line. If you search GitHub, the following is quite common:

$ cat Dockerfile
FROM rhel7/etcd
...

The short name works because podman inspects /etc/containers/registries.conf to get a list of registries to search for the image. The list of registries is searched from left to right until a match for the image is found, and that image is pulled. 

Podman reports the fully qualified name of the image pulled in the output, which might allow the user to notice an unexpected image was pulled. However, it’s quite possible the user will get an image that they were not expecting. Similar to DNS, without specifying the full URL, the image pull is not deterministic.

If a user’s registries.conf file was configured such that docker.io came before registry.access.redhat.com, and an attacker claimed the namespace rhel7 on hub.docker.com, and uploaded a malicious image to an ‘etcd’ repository in that namespace, pulling rhel7/etcd would result in downloading the attacker’s image, not the one published by Red Hat. 

If that image were then run, it would result in executing code from an attacker and possibly resulting in unintended consequences. Note similar potential problems could happen by using common typos. Say pulling rhel instead of specifying the version number.

The problem is compounded when there are multiple public registries in the search list. In RHEL 8.0 and 8.1, quay.io is listed before docker.io in the registries.conf file. If an attacker squatted on a popular docker.io image name on quay.io, a user on RHEL 8 could potentially pull an image from quay.io, which they probably intended to pull from docker.io. 

We want to thank Brad Geesaman for reporting this issue. His research was based on that of Rory McCune. In RHEL 8.2, the default registry search list is  changed to only include the curated Red Hat registries listed below. If you’re using a version of RHEL 8 before 8.2, the recommendation is to change the registry search list to only contain these registries:

$ cat /etc/containers/registries.conf
[registries.search]
registries = ['registry.access.redhat.com', 'registry.redhat.io', ‘docker.io’]

The problem also extends to container image tags. If a registry has a container repository with a matching name, but without a matching tag, the search continues to the next configured registry. If the next registry contains the image name and tag, the image is pulled. 

To demonstrate, I have registered the "trusted" namespace at both docker.io and quay.io. Then, I uploaded similar images to both registries in the “poc” repository, using different tags. Now, when trying to pull the image, notice that when the tag isn’t found in quay.io it’s pulled from docker.io:

$ cat /etc/containers/registries.conf
[registries.search]
registries = ['quay.io', 'docker.io']

Executing podman pull with the :docker tag pulls from docker.io. 

$ podman pull trusted/poc:docker
Trying to pull quay.io/trusted/poc:docker...
  manifest unknown: manifest unknown
Trying to pull docker.io/trusted/poc:docker...
...

While pulling the quay tag is pulled from quay.io:

$ podman pull trusted/poc:quay
Trying to pull quay.io/trusted/poc:quay...
...

If a popular container image on docker.io failed to upload a popular tag such as latest, an unsuspecting user could unintentionally pull that container image from an attacker’s repository on quay.io. Alternatively, if an attacker with adjacent network access was able to deny access to a popular image on docker.io, the same thing could happen. Pulling images when using a fully qualified image name does not do the same searching for tags.

What are we doing about it for Red Hat Enterprise Linux?

As of RHEL 8.2 beta and RHEL 7.8.z, the default registry search list is updated to the following values. To monitor the progress of the fixes, follow this issue for 8.2, and this issue for 7.8.z.

RHEL 8.2
$ cat /etc/containers/registries.conf
[registries.search]
registries = ['registry.access.redhat.com', 'registry.redhat.io', ‘docker.io’]

RHEL 7.8
$ cat /etc/containers/registries.conf
[registries.search]
registries = ['registry.access.redhat.com', 'registry.redhat.io', ‘docker.io’]

We decided to include docker.io in the default search list in RHEL to maintain backward compatibility with previous RHEL versions. However, we recommend removing it from the list if you don’t have existing applications that expect to be able to pull images by short name from that registry. See the RHEL 8 documentation for further details on changing the default registry search order.

If you would like to disable the use of short names on RHEL remove all registries from registries.conf.

What if I’m using OpenShift?

In OpenShift Container Platform 4, the machine-config-operator sets the default registry search list to include ‘registry.access.redhat.com’ and ‘docker.io’ similar to how we are configuring RHEL 7.8. During a build inside OpenShift 4, only docker.io is used. However we plan to add a feature to OpenShift 4 that would make searching of Red Hat registries more practical by sharing registry credentials to developer builds. 

In a future version of OpenShift 4 we might only include Red Hat registries in the default search list. Until such a feature is implemented, we recommend only using fully qualified image names in pods, and build configurations to avoid unintended images from being downloaded from docker.io.

On OpenShift 3.11, image short names will be searched using the following list, whether you’re using the default container runtime (Docker), or CRI-O.

$ cat /etc/containers/registries.conf
[registries.search]
registries = ["registry.redhat.io", "docker.io"]

Verifying the integrity of container images

Images uploaded to the Red Hat container registries are cryptographically signed and can be verified by changing some configuration options in RHEL or OpenShift. Future versions of RHEL and OpenShift may implement this verification by default. At this time, the verification is manually configured.

The configuration options mentioned in the linked articles only verify images that are pulled from Red Hat registries so it wouldn’t prevent the attack scenarios described in this article. To prevent this style of attack using image signatures, the default verification policy would need to be changed so that by default, images without signatures were rejected. In RHEL 8, issue the following command in addition to the ones mentioned in the article:

$ sudo podman image trust set -t reject default

After setting the default policy to reject, verify the setting using image trust show.

$ sudo podman image trust show
                             insecureAcceptAnything                                              
default                      reject                                                              
registry.access.redhat.com   signedBy security@redhat.com, security@redhat.com   
registry.redhat.io           signedBy security@redhat.com, security@redhat.com

Note that some deprecated repositories in the Red Hat registry don’t have signatures, so with a default reject policy those images will be rejected.

Conclusion

Red Hat recommends that users always use fully qualified names when referring to container images in any context. If using RHEL 8 before 8.2, update your registry search configuration to remove untrusted registries from the search list. 

Consider removing all registries from the search list so that short names don’t resolve. Enable container image signature verification for images pulled from Red Hat registries, and consider adjusting your default policy to reject unsigned images if you don’t need any images from registries other than Red Hat.