In early 2022, I had the privilege of writing a five-part series about Podman (see my Enable Sysadmin profile to find the articles). Podman is a daemonless engine for developing, managing, and running Open Container Initiative (OCI)-compliant containers. Containers aren't just a developer thing anymore; they're increasingly an intrinsic part of a sysadmin's work. Therefore it's important to know how to manage them in a practical and automated way.
[ Get the Podman basics cheat sheet. ]
This article shows one way you can use the power of Podman and systemd to create a container solution that starts and stops automatically with your operating system. I'll begin with why you might need something like this.
Use case scenario
Podman can run rootless containers, so you can run a container to do whatever you want, from completing a system task to running a full application solution, such as web servers or databases.
Say you want containers that run these services indefinitely. In a typical sysadmin environment, a privileged user configures these services to run at system boot and manages them with the systemctl
command. You can enable and start a service with the systemctl
command as a regular user. These system services start when the system boots and stop when the system shuts down.
When you invoke Podman at runtime from the command line, it runs only as long as you are in the session (graphical interface, text console, or SSH), and it stops when you close the session. This functionality is not ideal for your need to have a Podman-hosted service persist across reboots. Here's where integrating Podman and systemd is handy.
[ Get the systemd commands cheat sheet. ]
According to RHEL 9 documentation, with systemd unit files, you can:
- Set up a container or pod to start as a systemd service.
- Define the order in which the containerized services run and check for dependencies (for example, making sure another service is running, a file is available, or a resource is mounted).
- Control the state of the systemd system using the
systemctl
command. - Generate portable descriptions of containers and pods using systemd unit files.
Those are sample use cases. Here are the configuration steps.
[ Read how Podman 4.4 lets you make systemd better for Podman with Quadlet ]
Prepare the environment
First, as a privileged user, create a basic user in the system to act as a "service user" for the application with systemd:
# useradd webuser
# passwd webuser
Changing password for user webuser.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.
Keep this terminal open. In a separate terminal, SSH to the same server with the newly created user and verify it can run Podman commands:
$ ssh webuser@fedora
webuser@fedora's password:
Last login: Wed Feb 1 15:52:26 2023
$ podman version
Client: Podman Engine
Version: 4.3.1
API Version: 4.3.1
Go Version: go1.18.7
Built: Fri Nov 11 12:24:13 2022
OS/Arch: linux/amd64
Now, allow this "service user" account to start a service at system start that persists over logouts. Use the loginctl
command to configure the systemd user service to persist after the last user session of the configured service closes. In the privileged user's terminal window, do the following:
# loginctl show-user webuser | grep ^Linger
Linger=no
# loginctl enable-linger webuser
# loginctl show-user webuser | grep ^Linger
Linger=yes
Return to the basic user's terminal window, and pull the container image for your application. For this example, I will use the httpd container image because it's easier to demonstrate:
$ podman pull docker.io/library/httpd
Trying to pull docker.io/library/httpd:latest...
Getting image source signatures
Copying blob 70698c657149 done
Copying blob 00df85967755 done
Copying blob 8b4456c99d44 done
Copying blob ec2ee6bdcb58 done
Copying blob 8740c948ffd4 done
Copying config 6e794a4832 done
Writing manifest to image destination
Storing signatures
6e794a4832588ca05865700da59a3d333e7daaaf0544619e7f326eed7e72c903
$ podman image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/httpd latest 6e794a483258 2 weeks ago 149 MB
Run the container and mount an external volume in the system. If you've read my article on how to Create fast, easy, and repeatable containers with Podman and shell scripts (if you didn't, check it now), you're aware that I've already created this external volume with its contents in the /var/local/httpd/
directory:
$ ls /var/local/httpd/
index.html
$ cat /var/local/httpd/index.html
<html>
<header>
<title>Enable SysAdmin</title>
</header>
<body>
<p>Hello World!</p>
</body>
</html>
To make it easier (and since I'm a bit lazy), I will use the same content for this httpd container. Run it with Podman to check whether it works as intended:
$ podman run --name=httpd --hostname=httpd -p 8081:80 -v /var/local/httpd:/usr/local/apache2/htdocs:Z -d docker.io/library/httpd
f6f3d836bde9a75c6d745f481c8e83f8c7c342db2bbeac901347c535eaef03d9
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f6f3d836bde9 docker.io/library/httpd:latest httpd-foreground 5 seconds ago Up 4 seconds ago 0.0.0.0:8081->80/tcp httpd
$ curl -v http://localhost:8081
* Trying 127.0.0.1:8081...
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET / HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.82.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 01 Feb 2023 19:02:28 GMT
< Server: Apache/2.4.55 (Unix)
< Last-Modified: Fri, 07 Oct 2022 21:34:58 GMT
< ETag: "74-5ea7894e36d30"
< Accept-Ranges: bytes
< Content-Length: 116
< Content-Type: text/html
<
<html>
<header>
<title>Enable SysAdmin</title>
</header>
<body>
<p>Hello World!</p>
</body>
</html>
* Connection #0 to host localhost left intact
OK, it works. Stop the container:
$ podman stop httpd && podman rm -a && podman volume prune
httpd
f6f3d836bde9a75c6d745f481c8e83f8c7c342db2bbeac901347c535eaef03d9
WARNING! This will remove all volumes not used by at least one container. The following volumes will be removed:
No dangling volumes found
$ curl -v http://localhost:8081
* Trying 127.0.0.1:8081...
* connect to 127.0.0.1 port 8081 failed: Connection refused
* Trying ::1:8081...
* connect to ::1 port 8081 failed: Connection refused
* Failed to connect to localhost port 8081 after 1 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 8081 after 1 ms: Connection refused
The prerequisites are complete. It's time to move on to the next part.
[ Download now: A sysadmin's guide to Bash scripting. ]
Configure containers with systemd
First, start your container again with Podman:
$ podman run --name=httpd --hostname=httpd -p 8081:80 -v /var/local/httpd:/usr/local/apache2/htdocs:Z -d docker.io/library/httpd
16df315d3b23f41a70d1d3ffc11315c967cd213a107961df970327e85e62286f
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
16df315d3b23 docker.io/library/httpd:latest httpd-foreground 4 seconds ago Up 4 seconds ago 0.0.0.0:8081->80/tcp httpd
If you are familiar with creating systemd service units, you can do it "handcrafted." But why trouble yourself? Use the power of Podman to do it for you. It looks like this:
$ podman generate systemd --new --files --name httpd
/home/webuser/container-httpd.service
$ ls
container-httpd.service
$ cat container-httpd.service
# container-httpd.service
# autogenerated by Podman 4.3.1
# Wed Feb 1 16:06:04 -03 2023
[Unit]
Description=Podman container-httpd.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers
[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm \
-f %t/%n.ctr-id
ExecStart=/usr/bin/podman run \
--cidfile=%t/%n.ctr-id \
--cgroups=no-conmon \
--rm \
--sdnotify=conmon \
--replace \
--name=httpd \
--hostname=httpd \
-p 8081:80 \
-v /var/local/httpd:/usr/local/apache2/htdocs:Z \
-d docker.io/library/httpd
ExecStop=/usr/bin/podman stop \
--ignore -t 10 \
--cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
-f \
--ignore -t 10 \
--cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all
[Install]
WantedBy=default.target
Cool, isn't it? You can see all the parameters available with a command by running podman generate systemd --help
.
Now that you've generated the systemd service file with Podman, you won't run it manually anymore. Stop it like this:
$ podman stop httpd && podman rm -a && podman volume prune
httpd
16df315d3b23f41a70d1d3ffc11315c967cd213a107961df970327e85e62286f
WARNING! This will remove all volumes not used by at least one container. The following volumes will be removed:
No dangling volumes found
The next step is to create a ~/.config/systemd/user/
directory to hold the container-httpd.service
file and reload the systemd daemon. You could also use the /etc/systemd/system
directory, but only a privileged user can copy the systemd service file to this directory. Here are the commands:
$ mkdir -p ~/.config/systemd/user/
$ cp -Z container-httpd.service ~/.config/systemd/user/
$ ls ~/.config/systemd/user/
container-httpd.service
$ systemctl --user daemon-reload
It's almost done. Start the container with systemctl
using the newly created systemd service and check if it's working properly:
$ systemctl --user start container-httpd.service
$ systemctl --user status container-httpd.service
● container-httpd.service - Podman container-httpd.service
Loaded: loaded (/home/webuser/.config/systemd/user/container-httpd.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2023-02-01 16:11:28 -03; 1min 13s ago
Docs: man:podman-generate-systemd(1)
Process: 4564 ExecStartPre=/bin/rm -f /run/user/1001/container-httpd.service.ctr-id (code=exited, status=0/SUCCESS)
Main PID: 4600 (conmon)
Tasks: 14 (limit: 4649)
Memory: 4.6M
CPU: 139ms
CGroup: /user.slice/user-1001.slice/user@1001.service/app.slice/container-httpd.service
├─ 4585 /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp --enable-ipv6 -c -e 3 -r 4 --netns-t>
├─ 4587 rootlessport
├─ 4592 rootlessport-child
└─ 4600 /usr/bin/conmon --api-version 1 -c 81b6c1c3125bd9a10ad3bb5f62534cc01caf6e070a948be6ce0afa924d1b2b1b -u 81b6c1c3125bd9a10ad3bb5f>
…output omitted…
It seems to be working. To prove that, check the running container with Podman and try to curl
your application again:
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
81b6c1c3125b docker.io/library/httpd:latest httpd-foreground 2 minutes ago Up 2 minutes ago 0.0.0.0:8081->80/tcp httpd
$ curl -v http://localhost:8081
* Trying 127.0.0.1:8081...
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET / HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.82.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 01 Feb 2023 19:14:32 GMT
< Server: Apache/2.4.55 (Unix)
< Last-Modified: Fri, 07 Oct 2022 21:34:58 GMT
< ETag: "74-5ea7894e36d30"
< Accept-Ranges: bytes
< Content-Length: 116
< Content-Type: text/html
<
<html>
<header>
<title>Enable SysAdmin</title>
</header>
<body>
<p>Hello World!</p>
</body>
</html>
* Connection #0 to host localhost left intact
Bingo! You can now configure it to start and stop with your system by enabling it with systemctl
:
$ systemctl --user enable container-httpd.service
Created symlink /home/webuser/.config/systemd/user/default.target.wants/container-httpd.service → /home/webuser/.config/systemd/user/container-httpd.service.
Put it to the test and reboot the system. If it's working correctly, when the system boots, the httpd container service should start using systemd. It should be active, enabled, and responsive. Confirm this with curl
:
$ uptime
16:16:47 up 36 min, 1 user, load average: 0,00, 0,03, 0,16
# reboot
$ systemctl --user is-active container-httpd.service
active
$ systemctl --user is-enabled container-httpd.service
enabled
$ curl -v http://localhost:8081
* Trying 127.0.0.1:8081...
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET / HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.82.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 01 Feb 2023 19:20:18 GMT
< Server: Apache/2.4.55 (Unix)
< Last-Modified: Fri, 07 Oct 2022 21:34:58 GMT
< ETag: "74-5ea7894e36d30"
< Accept-Ranges: bytes
< Content-Length: 116
< Content-Type: text/html
<
<html>
<header>
<title>Enable SysAdmin</title>
</header>
<body>
<p>Hello World!</p>
</body>
</html>
* Connection #0 to host localhost left intact
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c8a15a2853ac docker.io/library/httpd:latest httpd-foreground 3 minutes ago Up 3 minutes ago 0.0.0.0:8081->80/tcp httpd
It worked! You have a container configured with a web application running as if it were a standard systemd service. It remains persistent across reboots or terminal exits. The same procedure works for pods with a few minor changes, too.
But what if you no longer need that service? How do you remove its persistence? As a systemd service, you can stop and disable it like this:
$ systemctl --user stop container-httpd.service
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ systemctl --user disable container-httpd.service
Removed /home/webuser/.config/systemd/user/default.target.wants/container-httpd.service.
As a final test, reboot the system again and ensure the application is not up:
$ uptime
16:21:34 up 4 min, 1 user, load average: 0,02, 0,10, 0,06
# reboot
$ uptime
16:22:06 up 0 min, 1 user, load average: 0,29, 0,06, 0,02
$ systemctl --user is-active container-httpd.service
inactive
$ systemctl --user is-enabled container-httpd.service
disabled
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ curl -v http://localhost:8081
* Trying 127.0.0.1:8081...
* connect to 127.0.0.1 port 8081 failed: Connection refused
* Trying ::1:8081...
* connect to ::1 port 8081 failed: Connection refused
* Failed to connect to localhost port 8081 after 0 ms: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 8081 after 0 ms: Connection refused
Gone with the wind! I hope I demonstrated clearly how the configuration and administration of persistent containers with systemd works.
I recommend you read these articles from my Enable Sysadmin colleagues that expand the ideas in this article:
- How to run pods as systemd services with Podman
- How to use auto-updates and rollbacks in Podman
- Mastering systemd: Securing and sandboxing applications and services
- Running containers with Podman and shareable systemd services
Wrap up
The powerful integration between Podman and systemd allows sysadmins to configure a container with a complete application solution that starts and stops automatically with the operating system.
Managing containers based on systemd units is mainly useful for basic and small deployments that do not need to scale. For more sophisticated scaling and orchestration of many container-based applications and services, consider an enterprise orchestration platform based on Kubernetes, such as Red Hat OpenShift Container Platform. I hope this article helps you understand this topic, supports your Linux certification path, and adds to your general sysadmin knowledge.