7 fun Linux containers/image transports features

One cool thing about using container tools like Podman, Buildah, CRI-O, and Skopeo, is that they are based on the same library for pulling and pushing images, which is containers/image. I often describe the containers/image project as a library for copying images between different types of container storage. This storage can be at a container registry, in containers/storage, as file system objects like directories or tarballs, and even directly in the Docker daemon. The mechanism for moving these container images between the different types of container storage is called a transport. One of the least understood features of containers/image is the various transports that it supports.

Transports

The image is specified using a transport:ImageName format. If no transport is specified, the docker (container registry) transport is used by default in all tools other than Skopeo. Skopeo requires users to always set the transport.

One of the cool things that Docker did was invent the idea of a container registry, which is basically a web server that contains container images. Docker.io, quay.io, and Artifactory are all examples of container registries. The Docker engineering team defined a protocol for pulling and pushing these images from the container registries, which we refer to as the docker transport.

[ You might also enjoy: 12 Podman guides to get started with containers ]

docker://docker-reference

An image reference is stored in a remote container image registry. Example: quay.io/podman/stable:latest. The reference can include a hostname to a specific registry.

$ skopeo copy docker://docker.io/alpine dir:/tmp/alpine
$ podman run docker://registry.fedoraproject.org/fedora:latest echo hello

For Podman and Buildah, docker://is the default transport. It can be dropped for convenience:

$ podman pull registry.fedoraproject.org/fedora:latest

When using a short name for pulling an image like fedora, Podman and Buildah use the list of registries to create a Docker reference and then call it using the docker:// transport.

Containers/image also supports a great many other transports for copying and storing container images.

oci:path

The oci transport exports/imports the content of a container image into a local directory, assuming the image is compliant with the Open Container Image Layout Specification. The manifest and layer tarballs are exported into the directory as individual files.

$ skopeo copy oci:/tmp/myimage docker://registry.example.com/myimage
$ podman run oci:/tmp/fedora echo hello

dir

The dir transport dumps the content of a container image into a local directory, again assuming the image is compliant with the Docker image layout. The manifest, layer tarballs, and signatures are exported as individual files into the directory. This is a non-standardized format, primarily useful for debugging or noninvasive container inspection. It is very similar to the oci transport but stores the files using the legacy docker format.

$ podman save --format docker-dir fedora -o /tmp/fedora
$ podman run dir:/tmp/fedora echo hello

docker-archive

Another way that the Docker engineers developed for storing and transporting container images around is using tar. A tarball contains all of the files that make up a container image. These tarballs were originally created using the docker save and podman save commands. They could be loaded back into the container engine's local storage using docker load and podman load. The container/image library makes this archive format into a transport, allowing it to be used directly by other commands like podman run dir:/tmp/fedora.tag echo hello, or buildah from dir:/tmp/fedora.tar

$ skopeo copy docker://registry.fedoraproject.org/fedora:latest docker-archive:/tmp/fedora.img
$ podman run docker-archive:/tmp/fedora.img echo hello

Warning: Using docker-archive is almost never the right thing to do. The format is very wasteful for both storage and I/O. I would recommend that users run a temporary docker/distribution registry instead.

oci-archive

This format is similar to the docker-archive, but instead of being the legacy format, it will store a single image in OCI Format. The path is an image compliant with the Open Container Image Layout Specification in a directory at the specified path and labeled with a tag.

$ skopeo copy docker-archive:/tmp/fedora.img oci-archive:/tmp/fedora-oci.img
$ podman run oci-archive:/tmp/fedora-oci.img echo hello

docker-daemon

One of the coolest features of containers/image is support for the docker-daemon transport. Docker and Podman do not share the same storage. They cannot because Docker controls locking to its storage within the daemon. While Podman, Buildah, CRI-O, and Skopeo can share content, they use the file system via containers/storage.

Podman and the other tools can work with the docker-daemon storage indirectly, via the docker-daemon transport. This allows the container engines to directly access images previously pulled by the Docker daemon or images built by docker build. You can also move images created by Buildah and Podman directly into the Docker daemon.

Something like this:

podman run docker-daemon:alpine echo hello

Note: Podman is pulling the image out of the Docker daemon, storing the image in containers/storage, and then running the container. It is not using the Docker storage directly. To let Podman communicate with Docker, we have to run it as root since the Docker daemon requires root privileges.

$ sudo docker pull fedora
$ sudo podman run docker-daemon:docker.io/library/fedora echo hello

If you build an image via buildah, you can commit the image directly into the docker-daemon:

# ctr=$(buildah from fedora)
# mnt=$(buildah mount $ctr)
# install -m 0744 start.sh $/mnt
# buildah commit $ctr docker-damon:myimage

Note: While it is cool to use the docker-daemon transport like this, it is a last-resort feature with very large I/O and storage costs. Consolidating the process to run in one or the other storage/ecosystem is almost always better. For example, it would be better to pull the image out of the Docker daemon and then just use containers/storage for running it.

$ sudo podman pull docker-daemon:registry.fedoraproject.org/library/fedora
$ podman run fedora echo hello

containers-storage

The final "transport" is really not a transport at all, but more of a mechanism for storing an image in a local storage created using the containers/storage library. The format of docker-reference is described in detail in the Docker transport.

$ skopeo copy docker://registry.access.redhat.com/ubi8:latest containers-storage:ubi8:latest

Buildah and Podman default to storing their images in container storage:

$ podman image pull ubi8

The Podman command above expands short names for ubi8 to registry.access.redhat.com/ubi8:latest and then uses the docker:// transport to pull it from the registry.access.redhat.com. It then stores the content directly in containers/storage.

$ buildah push docker://quay.io/myrepo/myimage

The Buildah command above looks into the container storage and then pushes the image using the docker:// transport to the quay.io registry.

[ Getting started with containers? Check out this free course. Deploying containerized applications: A technical overview. ]

Conclusion

The containers/image library provides an excellent way of transferring container images between machines and different kinds of storage. It is integral to the container engines and provides some really cool features. Please delve more into the transports using the containers-transports man page.