What's new in Red Hat Enterprise Linux 8.5 Container Tools?

In this post:  Red Hat Enterprise Linux (RHEL) 8.5 is a major update since RHEL 8.4 (What’s new in RHEL Container Tools?) and covers all of the features which have been released since then. In this article, we’ll cover the following main items:

• Update to the latest versions of Podman (3.3), Buildah (1.22), and Skopeo (1.4)

• Continued support of the 2.0 and 3.0 stable streams.

• Verification of container image signatures is now default

• UBI is now published on DockerHub

• The Podman container image is GA (Podman in Podman)

• Cgroup2 supported with crun and runc

• Better performance of Rootless containers (overlayfs)

Latest Versions of Podman, Buildah, and Skopeo

In RHEL 8, we provide two main types of application streams for the container tools: one fast, one stable for 24 months. You can see the versions of Podman, Buildah, and Skopeo with a single command:

yum module list | grep container-tools

Output:

container-tools      rhel8 [d][e]    common [d] [i]                           Most recent (rolling) versions of podman, buildah, skopeo, runc, conmon, runc, conmon, CRIU, Udica, etc as well as dependencies such as container-selinux built and tested together, and updated as frequently as every 12 weeks.
container-tools      1.0             common [d]                               Stable versions of podman 1.0, buildah 1.5, skopeo 0.1, runc, conmon, CRIU, Udica, etc as well as dependencies such as container-selinux built and tested together, and supported for 24 months.                                 
container-tools      2.0             common [d]                               Stable versions of podman 1.6, buildah 1.11, skopeo 0.1, runc, conmon, etc as well as dependencies such as container-selinux built and tested together, and supported as documented on the Application Stream lifecycle page.    
container-tools      3.0             common [d]                               Stable versions of podman 3.0, buildah 1.19, skopeo 1.2, runc, conmon, etc as well as dependencies such as container-selinux built and tested together, and supported as documented on the Application Stream lifecycle page.

In the output, you can see the versions of Podman, Buildah, and Skopeo in the stable streams, but if you install the fast stream, you’ll notice that it has been updated to Podman 3.3, Buildah 1.22, and Skopeo 1.4.

• Docker Compose 1.X, using the API driver, now works with rootless Podman.

• The podman play kube command now supports Kubernetes liveness probes, which will be created as Podman health checks.

• Auto-updating containers can now be done with locally-built images, not just images hosted on a registry, by creating containers with the io.containers.autoupdate label set to local.

• Support for third-party devices in Podman by including support for the Container Device Interface (CDI) standard.

• The podman container checkpoint and podman container restore commands can now be used to checkpoint containers in pods and restore those containers into pods.

• The podman cp command can now copy files from one container into another directly e.g., podman cp containerA:/etc/hosts containerB:/etc/)

• The podman auto-update command has had its output reformatted - it is now much clearer what images were pulled and what containers were updated.

• A set of new commands has been added to manage secrets! The podman secret create, podman secret inspect, podman secret ls and podman secret rm commands have been added to handle secrets, along with the --secret option to podman run and podman create to add secrets to containers. The initial driver for secrets does not support encryption - this will be added in a future release.

• The podman build command now supports a new option, --secret, to mount secrets into build containers.

• The podman-generate-kube command can now be run on Podman named volumes (generating PersistentVolumeClaim YAML), in addition to pods and containers.

• The podman play kube command now supports Kubernetes secretRef YAML (using the secrets support from podman secret) for environment variables.

• The podman play kube command can now read in Kubernetes YAML from STDIN when - is specified as file name (podman play kube -), allowing input to be piped into the command for scripting

There’s a lot of upstream work going on to improve “podman machine” but this is experimental upstream work as of today, and is not supported in RHEL. 

Also, there’s a lot of improvement and investment in podman play kube and podman generate kube commands which continues to improve the user experience going back and forth between OpenShift/Kubernetes and RHEL/Container Tools. 

Stay tuned for more good stuff as we move toward Podman 4.0!

Continued Support for 2.0 and 3.0 Stable Streams

As of the RHEl 8.5 launch, the Container Tools 2.0 and 3.0 application streams are currently supported, while the 1.0 application stream was deprecated in May 2021. The Container Tools 2.0 application stream is supported until May 2022, and the 3.0 application stream is supported until April 2023. 

For the up to date information, please see the Application Streams Release Life Cycle page.

Default Verification of container image signatures

This change should improve supply chain security while being completely transparent to most users (and especially production servers). In RHEL 8.5, we’ve deployed a default security policy (`/etc/containers/policy.json`) which will verify the GPG signatures of any images pulled from registry.access.redhat.com or registry.redhat.io. Red Hat has produced these GPG signatures and published them for many years, but historically users had to configure Podman to verify them. Now, with the latest version of Podman on RHEL 8.5+ these signatures will be verified by default.

RHEL 9 Beta

You might have difficulty pulling container images for RHEL 9.0 Beta after upgrading your host to RHEL 8.5. This is likely because signature validation of images is enabled by default in RHEL 8.5 and the RHEL Beta key has not been installed on the system. This is easy to enable.

First, pull the RHEL Beta GPG Key over SSL:

wget -O /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta https://www.redhat.com/security/data/f21541eb.txt

Next, configure Podman to use this new Beta GPG key:

podman image trust set -f /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta registry.access.redhat.com/ubi9-beta
podman image trust set -f /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-beta registry.access.redhat.com/rhel9-beta

Next, verify the new changes:

image trust show

Notice that the Beta key is used only for the beta images, and the main Red Hat GPG key is used for the standard images. Finally, note that GPG signatures are not checked for from any other registry:

default                                accept                                                            
                                       insecureAcceptAnything                                            
registry.access.redhat.com/rhel9-beta  signedBy                security@redhat.com                       https://access.redhat.com/webassets/docker/content/sigstore
registry.access.redhat.com/ubi9-beta   signedBy                security@redhat.com                       https://access.redhat.com/webassets/docker/content/sigstore
registry.redhat.io                     signedBy                security@redhat.com, security@redhat.com  https://registry.redhat.io/containers/sigstore
registry.access.redhat.com             signedBy                security@redhat.com, security@redhat.com  https://access.redhat.com/webassets/docker/content/sigstore

Next, pull a RHEL 9 Beta container image on a RHEL 8.5 container host, and you will notice that it pulls without a warning, and also verifies the signature against the beta GPG key:

podman pull registry.access.redhat.com/ubi9-beta/ubi-micro

UBI is now published on DockerHub

Red Hat’s goal is to offer choice in sourcing containers from popular registries. In support of this goal, all four UBI base images (micro, minimal, standard, and init) are available on DockerHub today (Red Hat Brings Red Hat Universal Base Image to Docker Hub).

These images are updated once every six weeks, or asynchronously (within days or hours) if there is a Critical or Important security CVE. Signature verification of  images is limited to Red Hat sponsored registries.  We intend to expand this and similar supply chain security mechanism to our partners (e.g., Docker) in the future. 

The Podman Container Image is Generally Available

There are some exciting announcements with the Podman Container Image. We kept this container image as Tech Preview in RHEL 8.4 to improve the UX based on some work by Urvashi Mohnani and Dan Walsh. Now, we’re making it generally available with RHEL 8.5 because we think the UX is more refined. We’re also adding this container image to UBI 8 which means you can distribute and use this Podman image anywhere you want.

It can be found in the Red Hat Ecosystem Catalog on the UBI 8 product page.

This image is extremely easy to try out. Here’s an example of running Podman in Podman. First, run the outer container. Notice, we’re running this outer container as a regular user:

$ podman run -it –privileged registry.access.redhat.com/ubi8/podman

Once the image is pulled, and the container is running, run the inside container. Notice, that we’re running this inside container as the virtual root user:

# podman run -it registry.access.redhat.com/ubi8/podman

Now, we are running the cat command as root in a container, which is running as our user account in the first container. 

Running a container in a container is useful when you need to try out a different version of Podman (newer or older), run podman (or Buildah) in a container on OpenShift, or even run Podman (or Buildah) on a third party platform like a cloud CI/CD system. Here’s the output:

sh-4.4# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.5 (Ootpa)
sh-4.4# 

For more information on running root inside and outside of a container, check out this blog: Understanding root inside and outside a container. For more information on running containers in other containers check out these two articles: How to use Podman inside of a container and How to use Podman inside of Kubernetes written by Urvashi Mohnani and Dan Walsh.

Support for cgroup2

In RHEL 8.5 we now support running Podman with RHEL booted in Control Groups V2 (cgroup2) mode. You can choose to run these containers with either runc or crun as your needs require. The runc utility remains the default in RHEL 8, and Control Groups version 1 remains the default in RHEL 8.

Control Groups V1 isn’t usable in rootless mode, nor does it support hierarchies. Control Groups V1 also had confinement issues, if a process is allowed to modify the cgroup file system, it can break out of its cgroup confinement. This limited the value of Control Groups V1 with Podman and containers.

With cgroup2 and Podman, you get the ability to manage rootless cgroups (for regular user accounts). The newer cgroup2 also supports hierarchical controls, which means a user can be put into a specific cgroup and then can further divide their cgroup into individual containers. 

Better Performance with Native OverlayFS

Rootless containers is a place where Podman and the RHEL Container Tools are constantly improving. Because of some kernel backports in RHEL 8.5, as well as some wonderful work that the Podman team did, we can now offer native OverlayFS support and performance, even when running rootless containers. 

Historically, we’ve run rootless containers using an implementation of OverlayFS which runs in the userspace. With this new feature, you can take advantage of the performance improvements of OverlayFS running in the kernel. 

Note, you will need to clear your local container storage to take advantage of this feature. WARNING: this will delete all of your existing containers and container images. First, perform a system reset:

# podman system reset

The above command will give you a helpful warning and make sure your storage is indeed reset so that you can take advantage of native OverlayFS. Next, run a container: 

podman run -id registry.access.redhat.com/ubi9-beta/ubi-micro

Now, verify that there is no /usr/bin/fuse-overlayfs program running:

ps -ef | grep fuse

For more information on the upstream work that drove this feature, there’s a great blog by Giuseppe Scrivano and Dan Walsh:  Podman is gaining rootless overlay support

Final Notes

First, if you’ve been testing RHEL 9.0 Beta, you’ll notice that the versions of Podman, Buildah, and Skopeo are identical. This is because the AppStream channel in RHEL 8 and RHEL 9 are meant to be quite similar. The idea is that you’ll be able to upgrade much easier. 

You’ll notice this synchronization as RHEL 9 GA releases, and it is planned to continue until RHEL 8.10 when the versions of Podman, Buildah, and Skopeo will freeze. At this point, the RHEL 9 Container Tools Application Stream will be the source for the latest Container Tools.

Conclusion

As you can see, we continue to innovate and add new features to the Container Tools application stream in RHEL 8, as well as supporting stable releases for users who can’t absorb the change. This offers users the best of both worlds. 

We’ve improved security by enforcing image verification and look to continue improvements in this area over time. We’re continuing to invest in performance improvements like kernel native OverlayFS and continue to invest upstream in creative solutions to improve Rootless containers.

We have also expanded the ecosystem to DockerHub to meet developers where they are, and ease the life of operations teams by giving them trusted base images which are ready for production.

Come check out the latest versions of our container tools, and feel free to give us feedback!