Container migration with Podman on RHEL

Starting with the beta release of Red Hat Enterprise Linux 8.1 Podman offers the possibility to migrate running containers from one system to another, without losing the state of the applications running in the container. With the help of CRIU Podman is able to offer stateful container migration for some containers.

The following is an example how to use container migration to move a running container from one system to another. The container in the example is running Wildfly, and is used to demonstrate multiple use cases of container migration.

Red Hat named a Leader in the 2023 Gartner® Magic Quadrant™

Red Hat was positioned highest for ability to execute and furthest for completeness of vision in the Gartner 2023 Magic Quadrant for Container Management.

Read the report

The first use case is the obvious one. Take a running container, checkpoint it, transfer it to another system and restore it. Stateful container migration.

Another interesting approach to use Podman’s container migration feature is to use it as a way to quickly start up a container which requires some time to initialize. The following examples are using the Wildfly application server which requires about 8 seconds to start up and to load the application I deployed. 

Checkpointing this container and restoring it from the checkpoint only takes about 4 seconds. Once the initial version of the container is running and ready to answer client requests the container is checkpointed and can then be restored requiring only 50% of the initial startup time. The container can be restored multiple times on the same host or on different hosts.

Container migration example

Based on the helloworld Wildfly quick start example I created a minimal application which returns a number and increments it. On the next request the client gets the next higher number back. A really simple but stateful application. To start Wildfly with Podman it needs a few additional options:

# podman run -d \
-v /home/deployments:/opt/jboss/wildfly/standalone/deployments \
--tmpfs /tmp \
--tmpfs /opt/jboss/wildfly/standalone/log \
--tmpfs /opt/jboss/wildfly/standalone/configuration \
--tmpfs /opt/jboss/wildfly/standalone/configuration/standalone_xml_history \
jboss/wildfly

This already shows one of the biggest container migration limitations. Currently, as implemented in Podman, stateful container migration only works with containers which do not change their file-system. Everything which the container potentially changes during runtime needs to be on a tmpfs. (I already opened a pull request (3443) which implements container migration including all root file-system changes, which will make it unnecessary to mount directories as tmpfs—so this limitation may soon be resolved upstream.)

As mentioned, all directories which are modified by Wildfly have to be mounted as tmpfs. Additionally, a volume which contains the helloworld application is mounted into the container: -v /home/deployments:/opt/jboss/wildfly/standalone/deployments

After about 8 seconds, the container is ready to answer client requests. The first step is to get the IP address of the container:

# podman inspect -l --format "{{.NetworkSettings.IPAddress}}"
10.88.0.247
# curl 10.88.0.247:8080/helloworld/
0
# curl 10.88.0.247:8080/helloworld/
1

The container can now be checkpointed and the checkpoint can be exported:

# podman container checkpoint -l -e /tmp/chkpt.tar.gz
# scp /tmp/chkpt.tar.gz rhel08:/tmp

Once the checkpoint archive has been transferred to the other system (rhel08) the container can be restored on that system:

# podman container restore -i /tmp/chkpt.tar.gz
# podman inspect -l --format "{{.NetworkSettings.IPAddress}}"
10.88.0.247
# curl 10.88.0.247:8080/helloworld/
2

At this point the stateful container has been migrated from one system to another without losing its state. This is also an example for the first use case of the container migration feature.

As previously mentioned container migration can also be used to reduce startup time of containers which require a certain time to initialize. Using the checkpoint archive (chkpt.tar.gz) from above it is possible to restore the checkpointed multiple times. This restore can happen on any host:

# podman container restore -i /tmp/chkpt.tar.gz -n hello1
# podman container restore -i /tmp/chkpt.tar.gz -n hello2
# podman container restore -i /tmp/chkpt.tar.gz -n hello3

This time I am using the parameter -n (--name), with which I can tell Podman to restore the container from the checkpoint archive with a different name.

# podman ps -a --format "{{.ID}} {{.Names}}"
a8b2e50d463c hello3
faabc5c27362 hello2
2ce648af11e5 hello1

#️ podman inspect hello1 --format "{{.NetworkSettings.IPAddress}}"
10.88.0.248
#️ curl 10.88.0.248:8080/helloworld/
4
#️ podman inspect hello2 --format "{{.NetworkSettings.IPAddress}}"
10.88.0.249
#️ curl 10.88.0.249:8080/helloworld/
4
#️ podman inspect hello3 --format "{{.NetworkSettings.IPAddress}}"
10.88.0.250
#️ curl 10.88.0.250:8080/helloworld/
4

This way it is possible to quickly start up stateful replicas of the initially checkpointed container.

All shown examples are running on the beta release of Red Hat Enterprise Linux 8.1.

Recording example #1:

Recording example #2: