Exploring additional image stores in Podman

In this article, I demonstrate how to store container images on shares, permitting the images to be accessed over the network. I'll use the additionalimagestores feature to accomplish this goal. Let's start with some background.

In a previous blog post, I discussed the image stores feature and how container images could be shared over the network.

"Additional image stores are a cool feature that allows you to set up additional read-only stores of images. For example, you could set up an NFS share with many overlay container images and share them with all of your container engines via NFS. Then rather than requiring each node running a container engine to pull down huge images, they could use the image on the NFS store and start the container."

When we added this feature, we expected users to be mounting network shares containing a large number of images to be used within a container. Now we are finding many more use cases for using them within the host.

The key field we need to focus on in the configuration files is named additionalimagestores. The field can be found in the /etc/containers/storage.conf or $HOME/.config/containers/storage.conf files. To provide more container image stores, add directories on your local system or mounted from a remote system to this field.

man containers-storage.conf
…
[storage.options]
additionalimagestores=[]
Paths to additional container image stores. Usually, these are read-only and stored on remote network shares.

Let's consider this solution by using a real-world example that includes sharing the images among several users.

The issue

A recent issue on the Podman repo brought up the use of the additionalimagestores field.

"I would like to set up a read-only shared image storage that multiple podman can use it. How can I pull the images to the readonly storage?

I have this line in my storage.conf

additionalimagestores = ['/storage']

Now, how should I instruct the podman pull command to store the image in the "/storage" directory, which I want to use as a shared read-only storage? "

I determined that the user wanted to share the storage amongst multiple users on his system. In other words, the user would pull the image into the host, and then allow root and other users all the share the image.

The solution

With Podman, it is easy to pull images to different storage directories by using the --root option, so let's start with that.

man podman
…
--root=value
Storage   root  dir  in  which  data,  including  images,  is  stored  (default: "/var/lib/containers/storage"  for  UID  0, "$HOME/.local/share/containers/storage" for other users).  Default root dir is configured in /etc/containers/storage.conf.

For my examples, I have chosen to share the /var/lib/mycontainers system directory among the different container engines.

If I want to pull content to a different folder, I can simply execute the following Podman command:

# podman --root /var/lib/mycontainers pull fedora
Trying to pull registry.fedoraproject.org/fedora...
Getting image source signatures
Copying blob 1657ffead824 done  
Copying config eb7134a03c done  
Writing manifest to image destination
Storing signatures
eb7134a03cd6bd8a3de99c16cf174d66ad2d93724bac3307795efcd8aaf914c5

# podman --root /var/lib/mycontainers images
REPOSITORY                          TAG      IMAGE ID       CREATED       SIZE
registry.fedoraproject.org/fedora   latest   eb7134a03cd6   2 weeks ago   208 MB

I can now use this additional capacity within a container (assuming the container is configured to use additional stores). We ship a stable image with Buildah preconfigured to use the additional stores and volume mount the /var/lib/mycontainers directory at the /var/lib/shared mount point. I described this configuration in a previous blog.

# podman run -v /var/lib/mycontainers:/var/lib/shared:ro quay.io/buildah/stable buildah images
REPOSITORY                          TAG      IMAGE ID       CREATED       SIZE     R/O
registry.fedoraproject.org/fedora   latest   eb7134a03cd6   2 weeks ago   208 MB   true

Very cool! The next step is to attempt to use the additional store in my home directory.

The first thing I need to do is create a storage.conf in my home directory that points at the shared storage.

$ cat ~/.config/containers/storage.conf
[storage]
driver = "overlay"

[storage.options]
additionalimagestores = [ "/var/lib/mycontainers",]

[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"

Now let's see whether we can view the images.

$ podman images
Error: Could not get runtime: error opening "/var/lib/mycontainers/overlay-images/images.lock": permission denied

We are not quite there. The problem is the non-root user does not have permission to read the root owner's storage. I must make the content world-readable and executable. I'll set the permissions like this:

$ sudo chmod -R a+rx /var/lib/mycontainers

Note that if you configure the permissions this way, the content in these directories is not secret. Usually, this is not a problem. If the images were pulled from public registries, then a rootless user could have pulled them and examined the content.

Now that the permissions are configured, let's run the podman images command again:

$ podman images
REPOSITORY                         TAG        IMAGE ID         CREATED       SIZE       R/O
registry.fedoraproject.org/fedora   latest   eb7134a03cd6   2 weeks ago   208 MB   true

It works!

Notice that the podman images command adds another column indicating that Podman is using read-only images. If you try to remove these images with rootless Podman, you will fail.

If a user pulls a different image, it is stored in their read-write storage, and they can use both.

$ podman pull alpine
Trying to pull docker.io/library/alpine...
Getting image source signatures
Copying blob cbdbe7a5bc2a done  
Copying config f70734b6a2 done  
Writing manifest to image destination
Storing signatures
f70734b6a266dcb5f44c383274821207885b549b75c8e119404917a61335981a

$ podman images
REPOSITORY                                 TAG    IMAGE ID          CREATED      SIZE       R/O
registry.fedoraproject.org/fedora   latest   eb7134a03cd6  2 weeks ago   208 MB     true
docker.io/library/alpine                   latest   f70734b6a266   5 weeks ago   5.88 MB  false

Now, let's run a container on this image.

$ podman run registry.fedoraproject.org/fedora ls -latr /
total 12
drwxr-xr-x.   2 nobody nobody    6 Jan 28 18:30 srv
lrwxrwxrwx.   1 nobody nobody    8 Jan 28 18:30 sbin -> usr/sbin
drwxr-xr-x.   2 nobody nobody    6 Jan 28 18:30 opt
drwxr-xr-x.   2 nobody nobody    6 Jan 28 18:30 mnt
drwxr-xr-x.   2 nobody nobody    6 Jan 28 18:30 media
lrwxrwxrwx.   1 nobody nobody    9 Jan 28 18:30 lib64 -> usr/lib64
lrwxrwxrwx.   1 nobody nobody    7 Jan 28 18:30 lib -> usr/lib
drwxr-xr-x.   2 nobody nobody    6 Jan 28 18:30 home
…

Cool, I was able to run a container without having to pull the image. I could set up lots of users with this configuration, and they could all share the storage. One problem you might have noticed is that all of the image files are owned by the user nobody. The reason is that Podman is running the container within a user namespace. The root user (UID=0) is not mapped into the user namespace. Any UID that is not mapped into the user namespace is reported as the nobody user.

However, you can still write to the image files.

$ podman run registry.fedoraproject.org/fedora bash -c "ls -ld /root;chown root:root /root; ls -ld /root"
dr-xr-xr-x. 2 nobody nobody 196 May 15 06:48 /root
dr-xr-xr-x. 2 root root 4096 May 15 06:48 /root

Containers use the overlay file system, so when you write to a file or directory as root, it allows you to copy up the content into a directory the user uses, and it will be owned by the root of the container.

We still have some work to do to make this a good working solution, since if I pull another image into the additional store Podman resets the permissions back to being non-readable.

$ sudo podman --root /var/lib/mycontainers pull ubi8
Trying to pull registry.access.redhat.com/ubi8...
Getting image source signatures
Copying blob 58e1deb9693d done  
Copying blob 78afc5364ad2 done  
Copying config 8e0b0194b7 done  
Writing manifest to image destination
Storing signatures
8e0b0194b7e1142c76aa26d71264d3561fab1b039fd3c49d9083dc9d3f42068f

$ podman images
Error: Could not get runtime: open /var/lib/mycontainers/overlay-images/images.json: permission denied

$ sudo chmod -R a+rx /var/lib/mycontainers/
$ podman images
REPOSITORY                                 TAG     IMAGE ID          CREATED      SIZE         R/O
registry.fedoraproject.org/fedora   latest    eb7134a03cd6   2 weeks ago  208 MB       true
docker.io/library/alpine                   latest    f70734b6a266   5 weeks ago   5.88 MB   false
registry.access.redhat.com/ubi8    latest   8e0b0194b7e1   5 weeks ago   211 MB       true

Conclusion

Additional stores are a very powerful concept. Imagine storing thousands of images on a network share. You could share it with hundreds of nodes and users. All of the users could then use the images without having to pull them to each node and user account.

[ Getting started with containers? Check out this free course. Deploying containerized applications: A technical overview. ]