Subscribe to the feed

There are countless use cases for running containers, whether on a single node with Podman or in an orchestrated fashion in Kubernetes distributions like Red Hat OpenShift. A common task in such environments is to build containers. At first glance, it seems strange to build containers inside a container, but it is pretty convincing when you see how fast a CI/CD system powered by OpenShift can get this kind of work done.

Dan Walsh explained in a detailed blog how to run Buildah inside a container. He also showed how to use the official Buildah images from quay.io/buildah, which can integrate easily into build pipelines. In the meantime, the need to run container tools inside of containers has further increased, and more and more users have reached out to us and asked for a similar blog post on running Skopeo in a container. Welcome to this blog!

So let’s start with Skopeo.

What is Skopeo, and why did we develop it?

Skopeo was born by the desire to inspect images on a remote registry without having to download the entire image with all its layers. Before Skopeo was released, users had to pull the whole image, even if they just wanted to inspect some metadata. Over the years, Skopeo has evolved into a powerful tool to perform all kinds of tasks around container images, including copying images, signing images, converting images across different formats and layer compressions, or syncing remote registries.

When speaking about our container tools, namely Podman, Buildah, Skopeo, and CRI-O, I usually compare them to a set of specialized Swiss Army knives that combined can meet pretty much all container use cases. Podman is, without a doubt, the biggest of these knives.

Podman provides a lot of functionality, from building and running containers to auto-updating containers running in Systemd units. Skopeo is the smallest and lightest of the tools and specializes in one thing only: managing container images. At the time of writing, skopeo copy ships with 26 command-line options to control the image format (i.e., Docker versus OCI), whether an image shall be signed, decrypted or encrypted, or to control how the layers of the image shall be compressed (e.g., gzip or zstd). Managing container images can reach a complexity beyond a simple podman-pull and podman-push. Skopeo is meant to own these complex tasks.

Why would I want to run Skopeo in a container?

The first thing that comes to mind is Kubernetes. Kubernetes is omnipresent in the modern IT infrastructure, and more and more systems are built on top of it. If you run a CI/CD system inside of Kubernetes or use OpenShift to build your container images, you may need to distribute those images across different container registries. Skopeo is an ideal tool for such tasks! And if we want to integrate Skopeo into a Kubernetes workflow, we need to run it in a container.

But there are reasons beyond Kubernetes to run Skopeo inside a container. Maybe you are running an older OS on the host but want to run a bleeding-edge Skopeo to enjoy the latest features? A common restriction in HPC environments is that rootless users are not allowed to install packages on the host. With the increasing popularity of Podman in HPC, running the Skopeo container with Podman to perform specific tasks is just a few commands away.

How can I run Skopeo in a container?

As for Buildah, we provide an official Fedora-based container image at quay.io/skopeo/stable. With the entry point set to /usr/bin/skopeo, the container calls Skopeo directly by default, so we can conveniently run the container and specify arguments and options to Skopeo (podman run or skopeo inspect, for example).

As soon as we have pulled down the image via podman pull quay.io/skopeo/stable, we can start using it. In theory, there is no functional difference between a locally installed Skopeo and running Skopeo in a container. In practice, we need to keep in mind that running Skopeo in a container means that the container’s root filesystem is isolated from the host's root filesystem. Hence, if we want the host and container to share or copy files, we need to make that possible, for instance, by volume-mounting files and directories. Let’s explore what that means step by step.

Inspecting a Remote Image

$ podman run --rm quay.io/skopeo/stable inspect docker://quay.io/buildah/stable

{                                                                                       

    "Name": "quay.io/buildah/stable",                                                   

    "Digest": "sha256:52a993631e5f13f142c1de0ac19ed2188a1a4190bf4530ecaecb23bece0289c3",

[...],

    "Env": [

        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",

        "DISTTAG=f31-updates-candidatecontainer",

        "FGC=f31-updates-candidate",

        "FBR=f31-updates-candidate",

        "BUILDAH_ISOLATION=chroot"

    ]

}

As shown above, inspecting a remote image via the Skopeo container is not much different from doing it with a Skopeo binary on the host. We should just make sure to clean up the container via --rm, so it is removed after exit.

Dealing with credentials and authentication

Working with container registries commonly requires us to authenticate to access and alter data, whether it be pulling or pushing an image or even deleting one. Skopeo supports various ways to specify credentials. Let’s start with the command line way.

For instance, skopeo inspect allows for specifying credentials on the command line via the --creds USERNAME[:PASSWORD] flag. Inspecting a remote image against a locked registry is simple:

$ podman run --rm quay.io/skopeo/stable inspect --creds $USER:$PASSWORD docker://$IMAGE

So far, there is really no difference between running Skopeo locally or in a container. However, we are heading toward a first example where we need to remind ourselves that running a container is different. This first example is authenticating via a so-called authentication file (authfile).

When doing a skopeo login, Skopeo logs into the specified registry and stores the authentication token in an authfile. The authfile prevents us from repeatedly specifying credentials but to log in only once. That’s great for lazy folks like me, and it further prevents us from leaking credentials in the bash history or anywhere else in cleartext.

When running on the same hosts, all container tools such as Skopeo, Buildah, and Podman share the same files, but how can we share authfiles when running Skopeo in a container? To keep things short, let’s jump directly to the solution:

$ podman run --rm -v $AUTHFILE:/auth.json quay.io/skopeo/stable inspect docker://$IMAGE

The critical part of the above example is -v $AUTHFILE:/auth.json, where we are volume-mounting an authfile at /auth.json in the container. Skopeo can now access the authentication tokens in the authfile and get access to the registry. Note that we configured quay.io/skopeo/stable to load /auth.json if present.

Other Skopeo commands work similarly. For instance, skopeo-copy allows for specifying credentials on the command line for the source and destination image via the --source-creds and --dest-creds flags. It also reads the /auth.json authfile. If you want to specify separate authfiles for the source and destination image, you can use the --source-authfile and --dest-authfile flags and volume-mount the files from the host into the container, just as we did in the skopeo inspect example.

Dealing with other configuration files

Skopeo and its sibling projects Buildah, Podman, and CRI-O, all share large parts of their code, with much of it being built on top of the containers/image library. The image library is incredibly flexible and has all kinds of options that we can change in various configuration files and directories. One example is the authfile, which we already took a detailed look at. In the following section, we want to mention further configuration files and directories you may want to consider for your specific needs:

  • Containers-certs.d: a directory to store custom TLS configurations for container registries. You can either volume-mount a directory at the default /etc/containers/certs.d path or use the --cert-dir command-line flags.
  • Containers-registries.conf: a file for configuring container registries. It allows for declaring certain registries as insecure (i.e., by disabling TLS verification), blocking them, or specifying per-registry mirrors. You can either volume-mount your configuration to /etc/containers/registries.conf, which overwrites the default configuration in the container, or you can volume-mount one or more files to /etc/registries.conf.d to extend the default configuration. Please refer to the registries.conf.d docs for more details.

Note that there are more configuration files and directories to consider. Using them follows the same pattern as above by volume-mounting the file or directory into the container's expected path. Further configuration files or directories to consider are containers-policy.json, containers-registries.d, containers-storage.conf, and generally any input files for Skopeo commands, such as the YAML files for skopeo-sync. Also note some of the host paths may vary depending whether the user is root or non-root.

How can I copy container images to or from the host?

Skopeo and its sibling projects all share the same local container-image storage. If Skopeo copies an image to the local storage, Podman, Buildah, and CRI-O can all use it. Hence, if we want to copy containers to or from the host’s container storage, we need to mount it into the Skopeo container.

The path to the host’s container storage varies between root (/var/lib/containers/storage) and non-root users ($HOME/.local/share/containers/storage). We need to keep that in mind. I love to enjoy the benefits of running rootless Podman, so let’s have a look at how I can share the container storage with the Skopeo container:

$ podman run --privileged --rm -v $HOME/.local/share/containers/storage:/var/lib/containers/storage quay.io/skopeo/stable copy docker://quay.io/buildah/stable containers-storage:quay.io/buildah/stable

Getting image source signatures

Copying blob sha256:5c1b9e8d7bf7b758fa84807a6bce45e4af333e1ddd566b5972550b6fcfbed9b8

Copying blob sha256:a4f5b2d92e7853576ab69fd69e994003840b13f96a103605d75020ce844bbf3a

Copying blob sha256:94fbb23a98101cd59d93a68d3a75b2d40a7336636a39f5b8f2ee83931f618402

Copying blob sha256:75d26b70c7ccaf987934812f4bee03ae071682ec41d6fb2c30316bb90a6e2811

Copying config sha256:865c44c0082260e35053fef081c75f05061bbf22f2f9fe9abe57feb7d5d2cf1e

Writing manifest to image destination

Storing signatures

That worked smoothly. We have now copied the quay.io/buildah/stable image into my local container storage, and podman images will list it!

But did you notice the --privileged flag? That flag disables all kinds of security features and container isolations that would stop the Skopeo container from accessing the mounted container storage. In particular, SELinux would prevent us from accessing the files, but we also need some capabilities (e.g., CAP_CHOWN, SYS_ADMIN). We usually do not recommend running with --privileged, but in this example, we want the trusted container to access the host’s container storage. With that being said, mounting the storage in such a way requires us to disable certain security mechanisms. We only recommend doing that in trusted environments.

A very pragmatic approach that avoids disabling security features is to export the images to a tarball (or any other path-based image transport) and mount them into the Skopeo container:

$ podman save --format oci-archive -o oci.tar $IMAGE

$ podman run --rm -v oci.tar:/oci.tar quay.io/skopeo/stable copy oci-archive:/oci.tar $DESTINATION

Conclusion

Running Skopeo in a container in Kubernetes, Podman, or even Docker, is easy. Skopeo is a very flexible tool for managing and distributing container images with dozens of options and supported transports. As shown in this blog, we can easily work with remote container registries, securely handle authentication, and access the host’s container storage with careful setup.

[ Getting started with containers? Check out this free course. Deploying containerized applications: A technical overview. ]


About the author

Container engineer at Red Hat, bass player, music lover.

Read full bio
UI_Icon-Red_Hat-Close-A-Black-RGB

Browse by channel

automation icon

Automation

The latest on IT automation for tech, teams, and environments

AI icon

Artificial intelligence

Updates on the platforms that free customers to run AI workloads anywhere

open hybrid cloud icon

Open hybrid cloud

Explore how we build a more flexible future with hybrid cloud

security icon

Security

The latest on how we reduce risks across environments and technologies

edge icon

Edge computing

Updates on the platforms that simplify operations at the edge

Infrastructure icon

Infrastructure

The latest on the world’s leading enterprise Linux platform

application development icon

Applications

Inside our solutions to the toughest application challenges

Original series icon

Original shows

Entertaining stories from the makers and leaders in enterprise tech