Deploying a multi-container application using Podman and Quadlet

In How to "build once, run anywhere" at the edge with containers, I hint that version 4.4 of Podman includes a new tool called Quadlet that enables simple ways to integrate Podman and systemd. And in Make systemd better for Podman with Quadlet, Dan Walsh demonstrates how to use Quadlet to deploy containers. In this article, I describe all the different unit file types Quadlet supports and show how you can use them when deploying containers using Podman and systemd.

[ Get hands on with Podman in this tutorial. ]

Quadlet supports these unit file types:

• .container: Used to manage containers by running podman run
• .kube: Used to manage containers defined in Kubernetes YAML files by running podman kube play
• .network: Used to create Podman networks that may be referenced in .container or .kube files
• .volume: Used to create Podman volumes that may be referenced in .container files.

I will use a multi-container application demo to explain how these four file types are used. The demo is based on the Kubernetes example Deploying WordPress and MySQL with persistent volumes with a few changes. First, instead of using Kubernetes, it uses Podman and Quadlet. Second, it shows how to run multi-container pods and wraps the WordPress application with a proxy that handles TLS encryption.

The demo aims to show how to use all four file types supported by Quadlet. To that end, it deploys the database service using a .container file while using a .kube file to deploy the WordPress application. In real life, users generally use either .container or .kube files in their deployments.

Create secrets

The demo uses three Podman secrets that you create outside of Quadlet. These secrets host these values:

• TLS certificates for the Envoy image
• Database root password for the Kubernetes pod
• Database root password for the database container

The envoy-certificates secret holds the TLS certificate and private key for the HTTPS traffic. It is a Podman secret based on a Kubernetes secret that includes the following keys:

• certificate.key: private key
• certificate.pem: pubic certificate

If you don't have certificates to use, you can create a self-signed certificate for testing using this command:

$ openssl req -x509 -sha256 -nodes -days 365 \
    -newkey rsa:4096 -keyout certificate.key -out certificate.pem

To create the Podman secret, run:

$ kubectl create secret generic \
    --from-file=certificate.key \
    --from-file=certificate.pem \
    envoy-certificates \
    --dry-run=client \
    -o yaml | \
    podman kube play -

Please note that because of the way the data is consumed, whether you're using a .container or .kube file, the second and third secrets hold the same information but in a different way.

The mysql-root-password-kube secret holds the password for the MySQL server. It is a Podman secret based on a Kubernetes secret that includes the following key: - password. The WordPress application, deployed via a .kube file uses this secret.

The mysql-root-password-container secret also holds the password for the MySQL server. However, this secret is a pure Podman secret and holds only the value of the password. The database container deployed using a .container file uses this secret.

To create both secrets using the same value, run the following commands:

$ MYSQL_ROOT_PASSWORD=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 13)

$ kubectl create secret generic \
    --from-literal=password="${MYSQL_ROOT_PASSWORD}" \
    mysql-root-password-kube \
    --dry-run=client \
    -o yaml | \
    podman kube play -

$ echo -n "${MYSQL_ROOT_PASSWORD}" | \
    podman secret create mysql-root-password-container -

[ Related reading: Storing sensitive data using Podman secrets: Which method should you use? ]

Create a Podman network

To allow communication between the different containers, create a Podman network by using a .network file. For this example, name the file quadlet-demo.network. You'll reference this file later in the database .container and WordPress pod .kube files.

[Network]
Subnet=192.168.30.0/24
Gateway=192.168.30.1
Label=app=wordpress

The demo does not necessarily require a network setup, but using it provides some important capabilities:

• Setting the network subnet address and the gateway address
• Setting a label

[ Download now: Podman basics cheat sheet ]

Deploy the database server

In the Kubernetes demo, the database server requires a persistent volume that translates to a Podman volume created using a .volume file. For this example, use the name quadlet-demo-mysql.volume:

[Volume]

The demo does not require any additional configuration for the volume.

To run the database server, use the following .container file named quadlet-demo-mysql.container:

[Install]
WantedBy=default.target

[Container]
Image=docker.io/library/mysql:5.6
ContainerName=quadlet-demo-mysql
Volume=quadlet-demo-mysql.volume:/var/lib/mysql
Network=quadlet-demo.network
# Once 4.5 is released change this line to use the quadlet Secret key
PodmanArgs=--secret=mysql-root-password-container,type=env,target=MYSQL_ROOT_PASSWORD

The Container section sets the following keys:

• Image: Use the official mysql image.
• ContainerName: Set the name of the container allowing the network to find it using DNS.
• Volume: Set the source of the volume as the previously defined .volume file telling Quadlet to create a systemd dependency between the unit files.
• Network: Set the name of the network to the previously defined .network file instructing Quadlet to create a systemd dependency between the unit files.
• PodmanArgs: Use the value from the secret as the value of the target environment variable.

Please note that this demo uses Podman v4.4.x in which Quadlet does not yet support the Secret key. Once Podman 4.5 is released, you can replace the PodmanArgs key with the Secret key like this:

Secret=mysql-root-password-container,type=env,target=MYSQL_ROOT_PASSWORD

[ Learn about new container events and auditing features in Podman 4.4. ] 

Configure WordPress

As discussed before, you'll deploy the WordPress application using a Kubernetes YAML file. First, define the storage as a PersistentVolumeClaim in the quadlet-demo.yml file:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wp-pv-claim
  labels:
    app: wordpress
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

[ Get the YAML cheat sheet ]

Then, add the pod definition to the same file:

---
apiVersion: v1
kind: Pod
metadata:
  name: quadlet-demo
spec:
  containers:
  - name: wordpress
    image: docker.io/library/wordpress:4.8-apache
    env:
    - name: WORDPRESS_DB_HOST
      value: quadlet-demo-mysql
    - name: WORDPRESS_DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mysql-root-password-kube
          key: password
    volumeMounts:
    - name: wordpress-persistent-storage
      mountPath: /var/www/html
  - name: envoy
    image: docker.io/envoyproxy/envoy:v1.25.0
    volumeMounts:
    - name: config-volume
      mountPath: /etc/envoy
    - name: certificates
      mountPath: /etc/envoy-certificates
    env:
    - name: ENVOY_UID
      value: "0"
  volumes:
  - name: config-volume
    configMap:
      name: envoy-proxy-config
  - name: certificates
    secret:
      secretName: envoy-certificates
  - name: wordpress-persistent-storage
    persistentVolumeClaim:
      claimName: wp-pv-claim

This pod contains two containers with the following configuration:

• WordPress application:
  • Uses the official WordPress container image
  • Sets the environment variable WORDPRESS_DB_HOST to the name of the database container
  • Gets the database password from the mysql-root-password-kube secret
  • Uses the wp-pv-claim persistent volume configured in the same YAML file as storage
• Envoy Proxy
  • Uses the official Envoy Proxy container image
  • Gets the TLS certificates from the envoy-certificates secret
  • Gets the configuration from the envoy-proxy-config ConfigMap

You can configure the Envoy Proxy using a ConfigMap defined in a separate YAML file named envoy-proxy-configmap.yml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-proxy-config
data:
  envoy.yaml: |
    admin:
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 9901
    static_resources:
      listeners:
      - name: listener_0
        address:
          socket_address:
            address: 0.0.0.0
            port_value: 8080
        filter_chains:
        - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_http
              codec_type: AUTO
              route_config:
                name: local_route
                virtual_hosts:
                - name: local_service
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: "/"
                    route:
                      cluster: backend
              http_filters:
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                - certificate_chain:
                    filename: /etc/envoy-certificates/certificate.pem
                  private_key:
                    filename: /etc/envoy-certificates/certificate.key
      clusters:
      - name: backend
        connect_timeout: 5s
        type: STATIC
        dns_refresh_rate: 1800s
        lb_policy: ROUND_ROBIN
        load_assignment:
          cluster_name: backend
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: 127.0.0.1
                    port_value: 80

Once you have all the pod configuration files ready, you can tie them all up together using a Quadlet .kube file, quadlet-demo.kube:

[Install]
WantedBy=default.target

[Unit]
Requires=quadlet-demo-mysql.service
After=quadlet-demo-mysql.service

[Kube]
# Point to the yaml file in the same directory
Yaml=quadlet-demo.yml
# Use the quadlet-demo network
Network=quadlet-demo.network
# Publish the envoy proxy data port
PublishPort=8000:8080
# Publish the envoy proxy admin port
PublishPort=9000:9901
# Use the envoy proxy config map in the same directory
ConfigMap=envoy-proxy-configmap.yml

The Kube section sets the following keys:

• Yaml: Set the Kubernetes YAML file.
• Network: Set the name of the network to the previously defined .network file instructing Quadlet to create a systemd dependency between the unit files
• PublishPort: Set the TCP port mapping for each container.
  • The application port 8080 in the WordPress container is published on port 8000.
  • The Envoy admin port 9901 is published on port 9000.
• ConfigMap: Load the ConfigMap defined in the additional Kubernetes YAML file.

Note that Quadlet supports paths relative to the location of the unit file. So, the values of the Yaml and ConfigMap keys point to files that reside in the same directory as the unit file.

In addition, since the WordPress application requires the database service, this unit depends on the service created by the .container unit. To achieve this, this unit file sets a Requires dependency to the corresponding database service unit instead of the container unit.

Deploy the application

Quadlet is a systemd generator installed as part of the Podman package. As a result, once Podman is installed, you can start using Quadlet.

To deploy your application using Quadlet:

1. Copy the unit files along with any additional files (such as the Kubernetes YAML file when using a .kube file) to the following directory:
   1. Rootful mode: /etc/containers/systemd
   2. Rootless mode: $HOME/.config/containers/systemd/
2. Force the generator by calling:
   1. Rootful mode: systemctl daemon-reload
   2. Rootless mode: systemctl –user daemon-reload
3. Start the services by calling:
   1. Rootful mode: systemctl start <Unit File Name>.service
   2. Rootless mode: systemctl –user start <Unit File Name>.service

For example, to deploy this demo in rootless mode, copy all previously defined files to $HOME/.config/containers/systemd/, then start the quadlet-demo service:

$ mkdir -p $HOME/.config/containers/systemd/
$ cp envoy-proxy-configmap.yml \
     quadlet-demo.kube \
     quadlet-demo-mysql.container \
     quadlet-demo-mysql.volume \
     quadlet-demo.network \
     quadlet-demo.yml \
     $HOME/.config/containers/systemd/
$ systemctl --user daemon-reload
$ systemctl --user start quadlet-demo.service

Note: If something goes wrong, systemd may not be able to tell you what is wrong with your unit file. You can use /usr/libexec/podman/quadlet --dryrun to see if there is an issue in the unit file.

Once you deploy the demo, you can test it by browsing to the WordPress application: https://<Machine FDQN or IP>:8000

Note that since the demo uses a self-signed certificate, you need to proceed to an unsafe connection. You may also browse to the Envoy Proxy admin page: 
http://<Machine FDQN or IP>:9000

Reproduce the demo with Ansible

You can find all the code for the demo on the quadlet-demo GitHub page. Follow the instructions in the README to deploy the demo on your environment using Ansible.

[ Learn how to build a Grafana dashboard to visualize data using Ansible and Podman. ]

Conclusion

Deploying container-based applications using Podman and systemd is simpler with Quadlet. Quadlet allows deploying workloads easily based on Kubernetes YAML files. You can use the same structured language to run containerized applications in Kubernetes or Red Hat OpenShift as well as edge devices without the overhead of Kubernetes. Quadlet .volume and .network files make configuring volumes and networks for your containers launched using Quadlet .container and .kubernetes files easier. This article gives you examples to follow and expand based on your use cases.

[ Visit the Podman topic page to keep up with the evolution of this container technology. ]