Podman at the edge: Keeping services alive with custom healthcheck actions

Podman is well known for its tight integration with systemd. Running containerized workloads in systemd is a simple yet powerful means for reliable and rock-solid deployments. The systemd integration further lays the foundation for more advanced Podman features such as auto updates and rollbacks or running Kubernetes workloads in systemd. This integration allows systemd to manage service dependencies, monitor the lifecycle and service state, and possibly also restart services in case of failure.

[ Get the systemd commands cheat sheet. ]

Such robust, self-healing, and self-updating deployments are crucial for edge deployments, where distributed and partially disconnected nodes must operate independently at times. We are excited to release a new custom health-check actions feature in Podman v4.3. This feature further expands Podman's capabilities in edge computing use cases.

Podman has supported healthchecks to identify degraded containers for some time, but there has been no way to correct them automatically. Healthcheck actions allow users to configure how Podman will react once a container turns unhealthy. For instance, an unhealthy container can be restarted automatically. This article explores how healthchecks work and how to set such custom actions.

What are healthchecks?

As the name suggests, healthchecks are used to check the health of a workload inside a container. If the healthcheck succeeds, the container is marked as "healthy"; otherwise, it will be "unhealthy." You can compare a healthcheck with running podman exec and examining the exit code.

Imagine you have a database running inside a container. A proper healthcheck tests whether the database is running and functioning correctly. The healthcheck could, for instance, perform a number of queries to the database and run further workload-specific tests. If the healthcheck has an exit code of zero, the container is "healthy."

[ Get the Podman basics cheat sheet. ]

Healthchecks can be set when building an image using the HEALTHCHECK instruction in the Dockerfile or when creating the container on the command line. The healthcheck status is reflected when inspecting containers using podman inspect or when listing containers with podman ps, which is extremely useful for monitoring the health of containers.

Until the release of Podman v4.3, monitoring was all you could do with healthchecks. There was no way to react once a container turned unhealthy. Hence, external tools had to poll the health states of containers to, for instance, restart unhealthy containers.

Specifying custom healthcheck actions

Starting with Podman v4.3, you can specify custom healthcheck actions to be executed once a container turns unhealthy. There are four such actions:

• restart: Podman restarts the container.
• stop: Podman stops the container.
• kill: Podman kills the container.
• none: Podman takes no action. This is used by default.

The "kill" action integrates well with systemd. When a Podman container runs inside a systemd unit, the container can use systemd's integrated restart policies. In case of a failure, for instance, when the container has a non-zero exit code, systemd detects that the container failed and automatically restarts the entire service, including the container. Similarly, setting the healthcheck on-failure action to "kill" will assign a non-zero exit code to a failed container, and systemd will restart the entire service. The "stop" action may behave differently, as a container can exit cleanly without a failure when it's stopped.

The "restart" option can be used outside of systemd and will help keep the container alive and healthy. The "none" action is the default to remain backwards compatible with previous versions of Podman. Thsee actions can be set using the command line's new -health-on-failure flag.

To put the pieces together, take a look at the following example.

First, create a container image with a healthcheck. To build the image, begin by creating a temporary directory, a script to run the healthcheck, and a script to serve as the entrypoint, which will wait until receiving SIGTERM and exit:

$ export TEMPDIR=$(mktemp -d)
$ cat >${TEMPDIR}/healthcheck <<EOF                                                      	 
#!/bin/sh                                                                                  	 
                                                                                            	 
if test -e /uh-oh; then                                                                    	 	 
 	exit 1                                                                                 	 
else                                                                                       	                                              	 
 	exit 0                                                                                 	 
fi                                                                                         	 
EOF
                                                                                            	 
 $ cat >${TEMPDIR}/entrypoint <<EOF                                                       	 
#!/bin/sh                                                                                  	 
                                                                                            	 
trap 'echo Received SIGTERM, finishing; exit' SIGTERM; echo WAITING; while :; do sleep 0.1; done
EOF

Now create a Dockerfile and copy the two scripts into the image.

$ cat >${TEMPDIR}/Dockerfile <<EOF                                                    	 
FROM registry.access.redhat.com/ubi9:latest                                                                                	 
                                                                                            	 
COPY healthcheck /healthcheck                                                              	 
COPY entrypoint  /entrypoint                                                               	 
                                                                                            	 
RUN  chmod 755 /healthcheck /entrypoint                                                                                            	 
CMD ["/entrypoint"]                                                                        	 
EOF
                                                                                            	 
$ podman build -t health-check-actions ${TEMPDIR}   

Next, use the health-check-actions image to create a container with the default action of "none":

$ podman run --replace -d --name test-container --health-cmd /healthcheck --health-on-failure=none --health-retries=1 health-check-actions

$ podman healthcheck run test-container


$ podman ps
CONTAINER ID  IMAGE                              	COMMAND  	CREATED     	STATUS                   	PORTS   	NAMES
90bb778b1a7d  localhost/health-check-actions:latest  /entrypoint  34 seconds ago  Up 34 seconds ago (healthy)          	test-container

The container is up and running and in a healthy state. So cause the healthcheck to fail by creating the /uh-oh file inside the container's root filesystem. The healthcheck will now fail, and the container will keep on running since no action will be taken.

$ podman exec test-container touch /uh-oh
$ podman healthcheck run test-container
unhealthy
$ podman ps
CONTAINER ID  IMAGE                              	COMMAND  	CREATED     	STATUS                     	PORTS   	NAMES
65e61ad28632  localhost/health-check-actions:latest  /entrypoint  20 seconds ago  Up 20 seconds ago (unhealthy)          	test-container

Recreate the container with the "kill" action. In this case, Podman should kill the container once it turns unhealthy.

$ podman run --replace -d --name test-container --health-cmd /healthcheck --health-on-failure=kill --health-retries=1 health-check-actions

$ podman ps
CONTAINER ID  IMAGE                              	COMMAND  	CREATED    	STATUS                  	PORTS   	NAMES
157aa6af4ee3  localhost/health-check-actions:latest  /entrypoint  7 seconds ago  Up 7 seconds ago (healthy)          	test-container

$ podman exec test-container touch /uh-oh
unhealthy

$ podman ps -a
CONTAINER ID  IMAGE                              	COMMAND  	CREATED     	STATUS                              	PORTS   	NAMES
157aa6af4ee3  localhost/health-check-actions:latest  /entrypoint  37 seconds ago  Exited (137) 9 seconds ago (unhealthy)          	test-container

Notice that the container has exited with 137. Combining the "kill" on-failure action with running Podman in systemd works really well to achieve rock-solid, self-healing workloads. systemd will restart the container automatically to turn the workload back into a healthy state.

Wrapping up

Being able to monitor your containers' health is crucial when you have containerized services running in production. This is even more critical when the services are in remote locations or critical systems.

Podman's tight integration with systemd lays the foundation for many use cases, such as auto updates and rollbacks or running Kubernetes workloads in systemd. Starting with Podman v4.3, you can further make use of custom health-check actions that, combined with running Podman in systemd, create an ideal environment for running containers in an edge computing environment.