How to use new container events and auditing features in Podman 4.4

Security-conscious users have been asking for ways to keep track of certain Podman events on the system. While Podman already had an elaborate event system where many actions, such as pulling an image or creating a container, are logged, the latest version, Podman 4.4, ships with a new feature that facilitates tracking security-related events better. This article discusses these Podman events and what you can do with them.

[ Download now: Podman basics cheat sheet ]

Access Podman events

Podman supports two backends for events: a file-based event system and one backed by journald/systemd. By default, Podman uses the journald backend as it is the central logging and journaling system for most Linux distributions. It is also straightforward for a system administrator to read all logs of all users from journald and stream them, for instance, to a central auditing server. Read How to find and interpret system log files on Linux for more insight on logs.) 

You can access Podman events directly with the podman events command. I'll run a simple Fedora container and see which events get triggered. Note that I pass the --since command-line flag to Podman to discard the events before creating the Fedora container.

$ now=$(date --iso-8601=seconds)
$ podman run -q --rm fedora echo "Hello, Podman v4.4"
Hello, Podman v4.4
$ podman events --since $now --stream=false
2023-02-07 11:35:06.551624026 +0100 CET container create 0bd9167e8746 (image=registry.fedoraproject.org/fedora:latest, name=condescending_nobel, license=MIT, name=fedora, vendor=Fedora Project, version=37)
2023-02-07 11:35:00.064559055 +0100 CET image pull  fedora
2023-02-07 11:35:06.607435527 +0100 CET container init 0bd9167e8746 (image=registry.fedoraproject.org/fedora:latest, name=condescending_nobel, name=fedora, vendor=Fedora Project, version=37, license=MIT)
2023-02-07 11:35:06.62205724 +0100 CET container start 0bd9167e8746 (image=registry.fedoraproject.org/fedora:latest, name=condescending_nobel, license=MIT, name=fedora, vendor=Fedora Project, version=37)
2023-02-07 11:35:06.622160621 +0100 CET container attach 0bd9167e8746 (image=registry.fedoraproject.org/fedora:latest, name=condescending_nobel, vendor=Fedora Project, version=37, license=MIT, name=fedora)
2023-02-07 11:35:06.622942873 +0100 CET container died 0bd9167e8746 (image=registry.fedoraproject.org/fedora:latest, name=condescending_nobel, vendor=Fedora Project, version=37, license=MIT, name=fedora)
2023-02-07 11:35:06.803541204 +0100 CET container remove 0bd9167e8746 (image=registry.fedoraproject.org/fedora:latest, name=condescending_nobel, name=fedora, vendor=Fedora Project, version=37, license=MIT)

A number of events happened underneath the podman run command:

• container create when creating a new container.
• image pull when pulling an image. That is because the fedora image was not present in the local storage, so Podman had to pull it down.
• container init when initializing the container in the runtime and doing some network setup.
• container start when starting the container. After that, the container is running.
• container attach when attaching to the terminal of a container. That is because I started the container in the foreground and did not detach from it.
• container died when the container exits.
• container remove because I used the --rm flag to remove the container once it exits.

There are many more events and far more resources than just containers, such as images, volumes, or networks. If you want to read more on these specific events, please refer to the man pages for details.

Look up events with journalctl

The default events backend in Podman is journald/systemd. In fact, running podman events just looks up journal entries matching specific keys and identifiers that Podman uses to store specific attributes of the events. For instance, PODMAN_NAME is the identifier for the name of the specific Podman resource (such as a container or image).

Instead of running podman events, you can use journalctl directly to look up Podman events. That can be a simple yet powerful way of looking at what users on the system have been up to.

I'll pull an image and use journalctl to look up the event:

$ podman pull -q hello-world
c62e0ebd9435f05c42bcc225fc395b1de40b2e7ee204f73fa073566a08e15caa
$ journalctl --user -r PODMAN_EVENT=pull PODMAN_NAME=hello-world
Feb 07 14:22:00 nebuchadnezzar podman[114843]: 2023-02-07 14:21:56.62938647 +0100 CET m=+0.024100372 image pull  hello-world
-- Boot 2924ec546cc242febe22417f932c4874 --
Dec 13 16:24:59 nebuchadnezzar podman[33229]: 2022-12-13 16:24:42.432159429 +0100 CET m=+0.044870186 image pull  hello-world

It seems I have downloaded the hello-world image twice. The first one just now and the other one on December 13, 2022, I assume, for testing purposes.

It is straightforward to query Podman events using journalctl. Filtering can be done by specifying the specific identifiers. Podman journald identifiers are documented on the man page, so please refer to it for details.

Create events for auditing

Security-conscious users have been making extensive use of Podman's event system for a while. Its integration with journald made it easy to figure out which user triggered which Podman events and when.

However, there was space for improvement. Events had to be connected to interpret them correctly. For example, the container-create event had to be linked with an image-pull event to know exactly which image was used. Furthermore, the container-create event did not include certain data users were asking for. For instance, they wanted to know which security settings a container was created with, such as SELinux labels, seccomp profiles, volumes, and mounts.

Podman 4.4 ships with a new feature that allows users to gather all relevant information about a container directly from a single container-create event. The attached data is the exact JSON payload from podman container inspect and includes a container's full configuration and security settings. An audit system may be watching the journal and looking for specific security settings and can notify the system administrator if something suspicious happens, such as if a container mounts specific paths on the host or receives too many privileges.

[ Download now: A system administrator's guide to IT automation. ]

I'll show how to enable verbose create events and extract the specific data with podman events and journalctl.

First, enable the new configuration option to instruct Podman to attach the container-inspect data. Since the data can be large and accumulate quickly, our team decided to make the new feature opt-in and hide it behind a configuration option in containers.conf. All my examples in this article are done as an ordinary rootless user, so I can configure the settings in my home directory in ~/.config/containers/containers.conf as follows:

[engine]                            	 
events_container_create_inspect_data=true

With the new feature enabled, I'll create a container and use podman events to look up the inspect data:

$ now=$(date --iso-8601=seconds)
$ podman create fedora
19524fe3c145df32d4f0c9af83e7964e4fb79fc4c397c514192d9d7620a36cd3
$ podman events --since $now --stream=false --no-trunc=false
2023-02-07 16:10:58.376361571 +0100 CET image pull  fedora
2023-02-07 16:10:58.43390195 +0100 CET container create 19524fe3c145 (image=registry.fedoraproject.org/fedora:latest, name=zen_wilson, license=MIT, name=fedora, vendor=Fedora Project, version=37)

The inspect data is not displayed by default. We decided to hide the data because the podman events output would otherwise render it unreadable for a human. However, you can explicitly demand this data as follows:

$ podman events --since $now --stream=false --format "{{.ContainerInspectData}}" | jq “.Config.CreateCommand”
[
  "/usr/bin/podman",
  "create",
  "fedora"
]

Using --format allows accessing specific fields of event data, so I'm exploiting it to look for the inspect data. For the sake of this example, I pipe the data to jq to display only the container's create command. The create command shows the exact command-line arguments the container is created with, which may be of interest for auditing.

Look for the same data but using journalctl:

$ journalctl --user -r PODMAN_EVENT=create --all -o json | jq ".PODMAN_CONTAINER_INSPECT_DATA | fromjson" | jq ".Config.CreateCommand"
[
  "/usr/bin/podman",
  "create",
  "fedora"
]

I get the exact same data. There are certainly more ways to access the data from the journal, but the important point is that the data is easily accessible and that there are solutions to process the journal and plug them into auditing.

Wrapping up

Knowing which containers are executed on a machine, what happened to them, and who did it is an important cornerstone of auditing. Podman 4.4 facilitates such auditing by aggregating all relevant data into a single event and journald entry that can easily be processed.

[ Kubernetes: Everything you need to know ]