New features for running containers on NFS with rootless Podman

Back in November, we wrote Rootless Podman on NFS, which explained why you cannot use the default setup of Podman when running with an NFS home directory. Since then, we have made it easier with newer versions of Podman and have discovered some issues with users setting up these environments.

Currently, rootless Podman (by default) stores its container image content in the user’s home directory. The blog linked above explains that Podman is blocked from using multiple UIDs on the NFS server side since the NFS server does not understand user namespace. The blog recommends that the user set up a different directory for storing content. Users need to edit the storage.conf file to make that happen.

About rootless_storage_path

We have added a new option field to storage.conf named rootless_storage_path to make it easier to use rootless Podman on an NFS home directory for different users.

$ man containers-storage.conf

       rootless_storage_path="$HOME/.local/share/containers/storage"

       Storage path for rootless users. By default, the graphroot for rootless users is set to $XDG_DATA_HOME/containers/storage, if XDG_DATA_HOME is set. Otherwise, $HOME/.local/share/containers/storage is used. This field can be used if administrators need to change the storage location for all users.

              The rootless storage path supports three substations:
              * `$HOME` => Replaced by the users home directory.
              * `$UID`  => Replaced by the users UID
              * `$USER` => Replaced by the users name

A common use case for this field is to provide a local storage directory when user home directories are NFS-mounted (podman does not support container storage over NFS).

This feature lets an administrator set up a machine with NFS home directories point the user’s container content to a separate directory.

For example, see the following setting:

rootless_storage_path="$/var/lib/user/$USER/containers/storage"

If the user sally executed Podman as a normal user, Podman would attempt to create the directories /var/lib/user/sally/containers/storage, and store the container images in that directory. It would also use these directories for all other containers. If ralph logged into the same machine and executed Podman, his storage would be /var/lib/user/ralph/containers/storage. Obviously, these directories would need to pre-exist and be writable by the respective user. The administrator could set up the parent directories, /var/lib/users, to be world-writable, and the Podman command would create the entire path.

Now, this leads me to my second point for this blog.

Using tmp directories for container storage

Administrators may be tempted to use /tmp or /var/tmp as container storage directories. Well, some of our users have reported problems when using these directories. The users reported that random files/directories disappeared from their images and containers. We diagnosed the problem and figured out that systemd was doing tmpfile cleanup. Systemd periodically triggers a job to clean up old content left in temporary directories. This job was erasing older files and directories stored in container-storage. The systemd-tmpfiles does not know about these Podman directories - it just does its job.

Wrap up

The bottom line is that if an administrator wants to store his content in /var/tmp or /tmp, they need to modify the systemd-tmpfiles to ignore the container content directories. Finally, on RHEL, CentOS, and Fedora, as well as many other distributions, the /tmp is on a tmpfs, which by default is only allowed to store up to 50% the size of memory, and will be destroyed on reboot.

[ Getting started with containers? Check out this free course: Deploying containerized applications: A technical overview. ]