Debugging image mode hosts

Image mode is a new approach to build, deploy, and manage Red Hat Enterprise Linux using bootable containers. In a previous blog post, we shared a bootable containers guide from the Fedora community to help you get started.

An important attribute of a bootable container is its immutability. An immutable operating system follows a different paradigm than traditional package-based systems. Immutability by default means security by design. Once deployed, the entire filesystem, with the exception of /etc and /var, is mounted read-only. This means that not even the root user has write privileges. Updates to the system are applied by downloading a new version of the bootable container image from a container registry, and then rebooting into the new state. It's a different way of approaching updates than using a package manager to update the system at runtime. It forces you to be intentional about changes to the operating system and gives you full state control.

Debugging an immutable OS

Production systems usually don't ship with debugging tools to keep the footprint as small as possible. On a traditional package-based system, you can use dnf to install strace for debugging purposes, but that doesn't exactly work in image mode. It's not sustainable to rebuild and reboot a bootable container image with debugging tools. Instead, you need a means to use debugging tools at runtime. Fortunately, there are two options.

Option 1: bootc usr-overlay

The bootc tool is the heart of Image Mode for RHEL and the core technology that enables OCI containers to encompass complete operating systems. Among other things, bootc is responsible for updating the operating system. It has a number of useful features for managing state and performing rollbacks. Additionally, it has functionality that's useful when debugging a system.

Suppose you want to install the strace command to help you debug some processes. You can use the bootc tool to temporarily unlock the operating system image by creating a transient writable overlayfs layer on /usr that gets discarded on reboot.

Open a terminal on the host system, and run the bootc command:

bootc usr-overlay

Now /usr is writable for root until the next reboot, which allows you to use dnf install strace for installation, just as you would on a traditional system.

Option 2: Toolbx

Image mode for RHEL is preinstalled with a development and troubleshooting tool called Toolbx. The command is available in the toolbox RPM, which we discussed in a previous article. It's particularly useful for installing troubleshooting tools at runtime without rebuilding the container image and then rebooting. For similar reasons, it is already popular on operating systems like Fedora CoreOS and Silverblue, which both have a similar design.

Toolbx enables you to use interactive command-line environments for software development, and troubleshooting the host operating system without having to install software on the host. It's built on top of Podman and other standard container technologies from OCI.

Toolbx environments have seamless access to the user’s home directory, Wayland and X11 sockets, networking (including Avahi), removable devices (like USB drives), systemd journal, SSH agent, D-Bus, ulimits, /dev, the udev database, and so on.

Toolbx installs software at runtime on immutable systems by providing a fully mutable container. In this container, you can install your favorite development and troubleshooting tools, editors, and SDKs. For example, it’s possible to do a dnf install -y strace without affecting the host operating system, and yet inspect the processes running on the host.

The Toolbx environment is based on an OCI image. On Red Hat Enterprise Linux, it defaults to the toolbox image from registry.access.redhat.com. This image is used to create a Toolbx container that offers the interactive command-line environment.

To get started, create a new container:

toolbox create

Then enter the environment:

toolbox enter

This presents an interactive command-line environment that looks and feels just like the one on the host operating system. The Toolbx commands are self-documenting. When you type an incomplete command, Toolbx provides documentation on what it expects next.

Toolbx can be used for a lot more than just strace. Everything from Ansible to Nmap to journalctl is possible, and it can be used both as your usual login UID and root. For more use cases and detailed examples, refer to the official Toolbx documentation.

More about bootable containers

If you're interested in bootable containers, we recommend taking a look at the upstream Getting Started Guide and Valentin's presentation on YouTube.