Painless services: implementing serverless with rootless Podman and systemd

Serverless is an event-driven computing paradigm where applications are allocated dynamically to serve a request or consume events. When the application is not in use, there are no computing resources allocated.

The serverless ecosystem offers a large number of runtimes, which start/stop/monitor software (e.g., Knative, Kubeless and many others). They come with different features, and they can trigger applications based on different kind of events (e.g., HTTP requests, messages, etc.).

Even if systemd cannot be considered a real serverless runtime, the socket activation feature provides a foundation for a serverless architecture.

Socket activation

The concept of systemd socket activation is that the socket is created by systemd, when a connection is received on such socket the application is activated. The application must be prepared to receive an already-initialized socket instead of creating a new one, for this reason not all applications can natively support socket activation.

To overcome this limitation, systemd provides a proxy supporting the socket activation and is capable of forwarding the connection to another socket.

The proxy

Recently (in version 209) systemd introduced a new tool named systemd-socket-proxyd. This is a handy tool that can be used with applications not supporting socket activation.

systemd-socket-proxyd is a proxy supporting socket activation that can be used to bi-directionally forward traffic from a socket inherited from systemd to another unix or network socket.

From the main page of systemd-socket-proxyd: One use of this tool is to provide socket activation support for services that do not natively support socket activation. On behalf of the service to activate, the proxy inherits the socket from systemd, accepts each client connection, opens a connection to a configured server for each client, and then bidirectionally forwards data between the two.

Since Podman does not support socket activation natively, systemd-socket-proxyd can be used as a sort of wrapper to intercept the socket activation and spin up a container.

The idea

Let's use a system with IP address 172.16.18.70 as an example. On this system we want to activate a rootless container running an httpd server exposing the HTTP service on port 8080.

Basically the flow is:

- systemd creates the TCP socket listening on 172.16.18.70:8080 using a .socket unit.

- Once the TCP socket is hit by a connection, the corresponding .service unit is activated. This service unit executes systemd-socket-proxyd and requires another .service unit that starts the container.

Why Podman?

Podman plays very well with systemd: here is a great video showing how podman can integrate with systemd to easily manage containerized services.

Podman can run rootless containers, and this post provides an overview of the advantages of rootless containers.

Let's do it

The following procedure has been tested on a Fedora 33 with systemd 246 and podman 2.2.1. The auto scale-down feature requires systemd 246, so it will not work on RHEL 8. However it's still worth understanding for future versions of RHEL.

As a first step we create the web server document root on the host and we put some dummy content: 

$ mkdir -p ~/www-data ; echo "Hello World" > ~/www-data/index.html

At this point we can create the httpd container. Please note that is exposing the port 8080 bound to the 127.0.0.1 address, and this is important to avoid conflicts with the systemd socket listening on the same port but on a different address.

$ podman pull registry.redhat.io/rhel8/httpd-24
$ podman create --name httpd -p 127.0.0.1:8080:8080 \
    -v ~/www-data:/var/www/html/:Z registry.redhat.io/rhel8/httpd-24

In order to define a user service we must create the systemd user units directory. From now we will work inside this directory:

$ mkdir -p ~/.config/systemd/user && cd ~/.config/systemd/user

As anticipated, podman can create the systemd unit files to manage the container, let's do it:

$ podman generate systemd -f -n httpd --new

The previous command generated a systemd unit file named container-httpd.service.

By default this unit file has an [Install] section that we don't need because the unit file should not be installed but activated only on-demand.

So we should edit the file and completely remove the [Install] section and all the contained directives.

Now we are ready to create the systemd socket, for that purpose we create an unit file named container-httpd-proxy.socket:

$ cat << EOF > container-httpd-proxy.socket
[Socket]
ListenStream=172.16.18.70:8080

[Install]
WantedBy=sockets.target
EOF

Please note that the socket is listening on 127.16.18.70:8080.  Leaving just the port number without the address will lead to a conflict with the container needing to open the port 8080.

At this point, we are ready to create the proxy service running systemd-socket-proxyd:

$ cat << EOF > container-httpd-proxy.service 
[Unit]
Requires=container-httpd.service
After=container-httpd.service
Requires=container-httpd-proxy.socket
After=container-httpd-proxy.socket

[Service]
ExecStart=/usr/lib/systemd/systemd-socket-proxyd 127.0.0.1:8080
EOF

Since this will be the unit activated by the socket, it is important to name this unit the same as the socket.

As you can see, the systemd-socket-proxyd forwards the connection to 127.0.0.1:8080 where the container is listening.

The other fundamental bit of this file is the Requires=container-httpd.service which is triggering the podman container when the unit is activated.

Here we are now ready to activate the socket unit, which is the only unit file that must be enabled at the start-up:

$ systemctl --user daemon-reload
$ systemctl --user enable --now container-httpd-proxy.socket
$ # let's check if the socket is open:
$ netstat -ltpn | grep 8080

Test it

After creating the systemd unit files and activating the socket, we should have a TCP socket listening on the 172.16.18.70:8080 address and the container httpd should not be running.

Now sending an HTTP request to http://172.16.18.70:8080 should trigger the container start:

$ curl http://172.16.18.70:8080
Hello World
$ podman ps
CONTAINER ID  IMAGE                              COMMAND               CREATED        STATUS            PORTS                     NAMES
c0cbdafc54ae  registry.redhat.io/rhel8/httpd-24  /usr/bin/run-http...  9 seconds ago  Up 8 seconds ago  127.0.0.1:8080->8080/tcp  httpd

Auto scale-down

With that, the container is started at the first request but is never terminated.

To avoid unneeded resource usage the container should be stopped when not in use.

To achieve that, there is a couple of useful features:

- systemd-socket-proxyd --exit-idle-time: permits to define a timeout to terminate in case of no connections. This command switch has been introduced in version 246.

- adding StopWhenUnneeded=yes to the container-httpd.service unit will stop the container when the proxy unit is terminated.

Putting all together

Let’s have a look to all the systemd units:

$ systemctl --user cat --no-pager container-httpd-proxy.socket 
[Socket]
ListenStream=172.16.18.70:8080

[Install]
WantedBy=sockets.target

$ systemctl --user cat --no-pager container-httpd-proxy.service 
[Unit]
Requires=container-httpd.service
After=container-httpd.service
Requires=container-httpd-proxy.socket
After=container-httpd-proxy.socket

[Service]
ExecStart=/usr/lib/systemd/systemd-socket-proxyd --exit-idle-time=30s 127.0.0.1:8080

$ systemctl --user cat --no-pager container-httpd.service 
[Unit]
Description=Podman container-httpd.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
StopWhenUnneeded=yes

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStartPre=/bin/rm -f %t/container-httpd.pid %t/container-httpd.ctr-id
ExecStart=/usr/bin/podman run --conmon-pidfile %t/container-httpd.pid --cidfile %t/container-httpd.ctr-id --cgroups=no-conmon -d --replace --name httpd -p 127.0.0.1:8080:8080 -v ${HOME}/www-data:/var/www/html/:Z registry.redhat.io/rhel8/httpd-24
ExecStartPost=/bin/sleep 1
ExecStop=/usr/bin/podman stop --ignore --cidfile %t/container-httpd.ctr-id -t 10
ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/container-httpd.ctr-id
PIDFile=%t/container-httpd.pid
KillMode=none
Type=forking

$ systemctl --user list-unit-files | grep container-httpd
container-httpd-proxy.service                                                          static    -            
container-httpd.service                                                                static    -            
container-httpd-proxy.socket                                                           enabled   disabled

Systemd opened the socket 172.16.18.70:8080:

$ netstat -ltpn | grep 8080
tcp  0  0 172.16.18.70:8080  0.0.0.0:*        LISTEN 12298/systemd

The `httpd` container is not running:

$ podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

Let’s send an HTTP request:

$ curl http://172.16.18.70:8080
Hello World

The request has been answered and the httpd container is started:

$ podman ps
CONTAINER ID  IMAGE                              COMMAND               CREATED        STATUS            PORTS                     NAMES
8487372aaadd  registry.redhat.io/rhel8/httpd-24  /usr/bin/run-http...  7 seconds ago  Up 6 seconds ago  127.0.0.1:8080->8080/tcp  httpd

Let’s wait 30 seconds and check if the container is stopped:

$ sleep 30 && podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

As we can see the container is not running anymore. Sending another request will trigger a new start of the container:

$ curl http://172.16.18.70:8080
Hello World

$ podman ps
CONTAINER ID  IMAGE                              COMMAND               CREATED        STATUS            PORTS                     NAMES
ebede0c85103  registry.redhat.io/rhel8/httpd-24  /usr/bin/run-http...  4 seconds ago  Up 3 seconds ago  127.0.0.1:8080->8080/tcp  httpd

Conclusion

Systemd and Podman can work very well together and thanks to the socket activation when a container is not serving any request can be stopped without keeping resources allocated.