Subscribe to the feed

Podman is well known for its tight integration with systemd. Running containerized workloads in systemd is a simple yet powerful means for reliable and rock-solid deployments. The systemd integration further lays the foundation for more advanced Podman features such as auto updates and rollbacks or running Kubernetes workloads in systemd. This integration allows systemd to manage service dependencies, monitor the lifecycle and service state, and possibly also restart services in case of failure.

[ Get the systemd commands cheat sheet. ]

Such robust, self-healing, and self-updating deployments are crucial for edge deployments, where distributed and partially disconnected nodes must operate independently at times. We are excited to release a new custom health-check actions feature in Podman v4.3. This feature further expands Podman's capabilities in edge computing use cases.

Podman has supported healthchecks to identify degraded containers for some time, but there has been no way to correct them automatically. Healthcheck actions allow users to configure how Podman will react once a container turns unhealthy. For instance, an unhealthy container can be restarted automatically. This article explores how healthchecks work and how to set such custom actions.

What are healthchecks?

As the name suggests, healthchecks are used to check the health of a workload inside a container. If the healthcheck succeeds, the container is marked as "healthy"; otherwise, it will be "unhealthy." You can compare a healthcheck with running podman exec and examining the exit code.

Imagine you have a database running inside a container. A proper healthcheck tests whether the database is running and functioning correctly. The healthcheck could, for instance, perform a number of queries to the database and run further workload-specific tests. If the healthcheck has an exit code of zero, the container is "healthy."

[ Get the Podman basics cheat sheet. ]

Healthchecks can be set when building an image using the HEALTHCHECK instruction in the Dockerfile or when creating the container on the command line. The healthcheck status is reflected when inspecting containers using podman inspect or when listing containers with podman ps, which is extremely useful for monitoring the health of containers.

Until the release of Podman v4.3, monitoring was all you could do with healthchecks. There was no way to react once a container turned unhealthy. Hence, external tools had to poll the health states of containers to, for instance, restart unhealthy containers.

Specifying custom healthcheck actions

Starting with Podman v4.3, you can specify custom healthcheck actions to be executed once a container turns unhealthy. There are four such actions:

  • restart: Podman restarts the container.
  • stop: Podman stops the container.
  • kill: Podman kills the container.
  • none: Podman takes no action. This is used by default.

The "kill" action integrates well with systemd. When a Podman container runs inside a systemd unit, the container can use systemd's integrated restart policies. In case of a failure, for instance, when the container has a non-zero exit code, systemd detects that the container failed and automatically restarts the entire service, including the container. Similarly, setting the healthcheck on-failure action to "kill" will assign a non-zero exit code to a failed container, and systemd will restart the entire service. The "stop" action may behave differently, as a container can exit cleanly without a failure when it's stopped.

The "restart" option can be used outside of systemd and will help keep the container alive and healthy. The "none" action is the default to remain backwards compatible with previous versions of Podman. Thsee actions can be set using the command line's new -health-on-failure flag.

To put the pieces together, take a look at the following example.

First, create a container image with a healthcheck. To build the image, begin by creating a temporary directory, a script to run the healthcheck, and a script to serve as the entrypoint, which will wait until receiving SIGTERM and exit:

$ export TEMPDIR=$(mktemp -d)
$ cat >${TEMPDIR}/healthcheck <<EOF                                                      	 
#!/bin/sh                                                                                  	 
                                                                                            	 
if test -e /uh-oh; then                                                                    	 	 
 	exit 1                                                                                 	 
else                                                                                       	                                              	 
 	exit 0                                                                                 	 
fi                                                                                         	 
EOF
                                                                                            	 
 $ cat >${TEMPDIR}/entrypoint <<EOF                                                       	 
#!/bin/sh                                                                                  	 
                                                                                            	 
trap 'echo Received SIGTERM, finishing; exit' SIGTERM; echo WAITING; while :; do sleep 0.1; done
EOF

Now create a Dockerfile and copy the two scripts into the image.

$ cat >${TEMPDIR}/Dockerfile <<EOF                                                    	 
FROM registry.access.redhat.com/ubi9:latest                                                                                	 
                                                                                            	 
COPY healthcheck /healthcheck                                                              	 
COPY entrypoint  /entrypoint                                                               	 
                                                                                            	 
RUN  chmod 755 /healthcheck /entrypoint                                                                                            	 
CMD ["/entrypoint"]                                                                        	 
EOF
                                                                                            	 
$ podman build -t health-check-actions ${TEMPDIR}   

Next, use the health-check-actions image to create a container with the default action of "none":

$ podman run --replace -d --name test-container --health-cmd /healthcheck --health-on-failure=none --health-retries=1 health-check-actions

$ podman healthcheck run test-container


$ podman ps
CONTAINER ID  IMAGE                              	COMMAND  	CREATED     	STATUS                   	PORTS   	NAMES
90bb778b1a7d  localhost/health-check-actions:latest  /entrypoint  34 seconds ago  Up 34 seconds ago (healthy)          	test-container

The container is up and running and in a healthy state. So cause the healthcheck to fail by creating the /uh-oh file inside the container's root filesystem. The healthcheck will now fail, and the container will keep on running since no action will be taken.

$ podman exec test-container touch /uh-oh
$ podman healthcheck run test-container
unhealthy
$ podman ps
CONTAINER ID  IMAGE                              	COMMAND  	CREATED     	STATUS                     	PORTS   	NAMES
65e61ad28632  localhost/health-check-actions:latest  /entrypoint  20 seconds ago  Up 20 seconds ago (unhealthy)          	test-container

Recreate the container with the "kill" action. In this case, Podman should kill the container once it turns unhealthy.

$ podman run --replace -d --name test-container --health-cmd /healthcheck --health-on-failure=kill --health-retries=1 health-check-actions

$ podman ps
CONTAINER ID  IMAGE                              	COMMAND  	CREATED    	STATUS                  	PORTS   	NAMES
157aa6af4ee3  localhost/health-check-actions:latest  /entrypoint  7 seconds ago  Up 7 seconds ago (healthy)          	test-container

$ podman exec test-container touch /uh-oh
unhealthy

$ podman ps -a
CONTAINER ID  IMAGE                              	COMMAND  	CREATED     	STATUS                              	PORTS   	NAMES
157aa6af4ee3  localhost/health-check-actions:latest  /entrypoint  37 seconds ago  Exited (137) 9 seconds ago (unhealthy)          	test-container

Notice that the container has exited with 137. Combining the "kill" on-failure action with running Podman in systemd works really well to achieve rock-solid, self-healing workloads. systemd will restart the container automatically to turn the workload back into a healthy state.

Wrapping up

Being able to monitor your containers' health is crucial when you have containerized services running in production. This is even more critical when the services are in remote locations or critical systems.

Podman's tight integration with systemd lays the foundation for many use cases, such as auto updates and rollbacks or running Kubernetes workloads in systemd. Starting with Podman v4.3, you can further make use of custom health-check actions that, combined with running Podman in systemd, create an ideal environment for running containers in an edge computing environment.


About the authors

Container engineer at Red Hat, bass player, music lover.

Read full bio

Matt Heon has been a software engineer on Red Hat's Container Runtimes team for the last five years. He's one of the original authors and lead maintainers of the Podman project. He focuses on container security, networking, and low-level development.

Read full bio

Preethi Thomas is an Engineering Manager for the containers team at Red Hat. She has been a manager for over three years. Prior to becoming a manager, she was a Quality Engineer at Red Hat. She is passionate about open source software, software quality, and open management practices and has rich experience working with upstream communities and projects. She is also highly passionate about Diversity and Inclusion and actively participates in outreach activities.

Read full bio
UI_Icon-Red_Hat-Close-A-Black-RGB

Browse by channel

automation icon

Automation

The latest on IT automation for tech, teams, and environments

AI icon

Artificial intelligence

Updates on the platforms that free customers to run AI workloads anywhere

open hybrid cloud icon

Open hybrid cloud

Explore how we build a more flexible future with hybrid cloud

security icon

Security

The latest on how we reduce risks across environments and technologies

edge icon

Edge computing

Updates on the platforms that simplify operations at the edge

Infrastructure icon

Infrastructure

The latest on the world’s leading enterprise Linux platform

application development icon

Applications

Inside our solutions to the toughest application challenges

Original series icon

Original shows

Entertaining stories from the makers and leaders in enterprise tech