Exploring additional image stores in Podman
In this article, I demonstrate how to store container images on shares, permitting the images to be accessed over the network. I'll use the additionalimagestores feature to accomplish this goal. Let's start with some background.
In a previous blog post, I discussed the image stores feature and how container images could be shared over the network.
"Additional image stores are a cool feature that allows you to set up additional read-only stores of images. For example, you could set up an NFS share with many overlay container images and share them with all of your container engines via NFS. Then rather than requiring each node running a container engine to pull down huge images, they could use the image on the NFS store and start the container."
When we added this feature, we expected users to be mounting network shares containing a large number of images to be used within a container. Now we are finding many more use cases for using them within the host.
The key field we need to focus on in the configuration files is named additionalimagestores. The field can be found in the
$HOME/.config/containers/storage.conf files. To provide more container image stores, add directories on your local system or mounted from a remote system to this field.
man containers-storage.conf … [storage.options] additionalimagestores= Paths to additional container image stores. Usually, these are read-only and stored on remote network shares.
Let's consider this solution by using a real-world example that includes sharing the images among several users.
"I would like to set up a read-only shared image storage that multiple podman can use it. How can I pull the images to the readonly storage?
I have this line in my storage.conf
additionalimagestores = ['/storage']
Now, how should I instruct the podman pull command to store the image in the "/storage" directory, which I want to use as a shared read-only storage? "
I determined that the user wanted to share the storage amongst multiple users on his system. In other words, the user would pull the image into the host, and then allow root and other users all the share the image.
With Podman, it is easy to pull images to different storage directories by using the
--root option, so let's start with that.
man podman … --root=value Storage root dir in which data, including images, is stored (default: "/var/lib/containers/storage" for UID 0, "$HOME/.local/share/containers/storage" for other users). Default root dir is configured in /etc/containers/storage.conf.
For my examples, I have chosen to share the
/var/lib/mycontainers system directory among the different container engines.
If I want to pull content to a different folder, I can simply execute the following Podman command:
# podman --root /var/lib/mycontainers pull fedora Trying to pull registry.fedoraproject.org/fedora... Getting image source signatures Copying blob 1657ffead824 done Copying config eb7134a03c done Writing manifest to image destination Storing signatures eb7134a03cd6bd8a3de99c16cf174d66ad2d93724bac3307795efcd8aaf914c5 # podman --root /var/lib/mycontainers images REPOSITORY TAG IMAGE ID CREATED SIZE registry.fedoraproject.org/fedora latest eb7134a03cd6 2 weeks ago 208 MB
I can now use this additional capacity within a container (assuming the container is configured to use additional stores). We ship a stable image with Buildah preconfigured to use the additional stores and volume mount the
/var/lib/mycontainers directory at the
/var/lib/shared mount point. I described this configuration in a previous blog.
# podman run -v /var/lib/mycontainers:/var/lib/shared:ro quay.io/buildah/stable buildah images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest eb7134a03cd6 2 weeks ago 208 MB true
Very cool! The next step is to attempt to use the additional store in my home directory.
The first thing I need to do is create a
storage.conf in my home directory that points at the shared storage.
$ cat ~/.config/containers/storage.conf [storage] driver = "overlay" [storage.options] additionalimagestores = [ "/var/lib/mycontainers",] [storage.options.overlay] mount_program = "/usr/bin/fuse-overlayfs"
Now let's see whether we can view the images.
$ podman images Error: Could not get runtime: error opening "/var/lib/mycontainers/overlay-images/images.lock": permission denied
We are not quite there. The problem is the non-root user does not have permission to read the root owner's storage. I must make the content world-readable and executable. I'll set the permissions like this:
$ sudo chmod -R a+rx /var/lib/mycontainers
Note that if you configure the permissions this way, the content in these directories is not secret. Usually, this is not a problem. If the images were pulled from public registries, then a rootless user could have pulled them and examined the content.
Now that the permissions are configured, let's run the
podman images command again:
$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest eb7134a03cd6 2 weeks ago 208 MB true
Notice that the
podman images command adds another column indicating that Podman is using read-only images. If you try to remove these images with rootless Podman, you will fail.
If a user pulls a different image, it is stored in their read-write storage, and they can use both.
$ podman pull alpine Trying to pull docker.io/library/alpine... Getting image source signatures Copying blob cbdbe7a5bc2a done Copying config f70734b6a2 done Writing manifest to image destination Storing signatures f70734b6a266dcb5f44c383274821207885b549b75c8e119404917a61335981a $ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest eb7134a03cd6 2 weeks ago 208 MB true docker.io/library/alpine latest f70734b6a266 5 weeks ago 5.88 MB false
Now, let's run a container on this image.
$ podman run registry.fedoraproject.org/fedora ls -latr / total 12 drwxr-xr-x. 2 nobody nobody 6 Jan 28 18:30 srv lrwxrwxrwx. 1 nobody nobody 8 Jan 28 18:30 sbin -> usr/sbin drwxr-xr-x. 2 nobody nobody 6 Jan 28 18:30 opt drwxr-xr-x. 2 nobody nobody 6 Jan 28 18:30 mnt drwxr-xr-x. 2 nobody nobody 6 Jan 28 18:30 media lrwxrwxrwx. 1 nobody nobody 9 Jan 28 18:30 lib64 -> usr/lib64 lrwxrwxrwx. 1 nobody nobody 7 Jan 28 18:30 lib -> usr/lib drwxr-xr-x. 2 nobody nobody 6 Jan 28 18:30 home …
Cool, I was able to run a container without having to pull the image. I could set up lots of users with this configuration, and they could all share the storage. One problem you might have noticed is that all of the image files are owned by the user nobody. The reason is that Podman is running the container within a user namespace. The root user (UID=0) is not mapped into the user namespace. Any UID that is not mapped into the user namespace is reported as the nobody user.
However, you can still write to the image files.
$ podman run registry.fedoraproject.org/fedora bash -c "ls -ld /root;chown root:root /root; ls -ld /root" dr-xr-xr-x. 2 nobody nobody 196 May 15 06:48 /root dr-xr-xr-x. 2 root root 4096 May 15 06:48 /root
Containers use the overlay file system, so when you write to a file or directory as root, it allows you to copy up the content into a directory the user uses, and it will be owned by the root of the container.
We still have some work to do to make this a good working solution, since if I pull another image into the additional store Podman resets the permissions back to being non-readable.
$ sudo podman --root /var/lib/mycontainers pull ubi8 Trying to pull registry.access.redhat.com/ubi8... Getting image source signatures Copying blob 58e1deb9693d done Copying blob 78afc5364ad2 done Copying config 8e0b0194b7 done Writing manifest to image destination Storing signatures 8e0b0194b7e1142c76aa26d71264d3561fab1b039fd3c49d9083dc9d3f42068f $ podman images Error: Could not get runtime: open /var/lib/mycontainers/overlay-images/images.json: permission denied
$ sudo chmod -R a+rx /var/lib/mycontainers/ $ podman images REPOSITORY TAG IMAGE ID CREATED SIZE R/O registry.fedoraproject.org/fedora latest eb7134a03cd6 2 weeks ago 208 MB true docker.io/library/alpine latest f70734b6a266 5 weeks ago 5.88 MB false registry.access.redhat.com/ubi8 latest 8e0b0194b7e1 5 weeks ago 211 MB true
Additional stores are a very powerful concept. Imagine storing thousands of images on a network share. You could share it with hundreds of nodes and users. All of the users could then use the images without having to pull them to each node and user account.
[ Getting started with containers? Check out this free course. Deploying containerized applications: A technical overview. ]