Crictl Vs Podman

As people continue to adopt CRI-O as a new container runtime for Kubernetes I am hearing questions from administrators who are confused whether they should use Crictl or Podman to diagnose and understand what is going on in a Kubernetes node. This is not one or the other -- these tools are complementary, and this article attempts to explain the tools and examine when it is best to use each of these tools. If you take away one thing from this post, remember that Crictl checks the front entrance, while Podman examines the foundation.

First things first.  For those people who aren’t familiar with it, CRI-O is a lightweight, Open Container Initiative (OCI) compliant, container runtime for Kubernetes.  It is designed to run any OCI-based container, it is optimized for Kubernetes and committed to being stable and conformant with the Kubernetes container runtime interface with each Kubernetes release.  CRI-O is also now fully supported in OpenShift, Red Hat’s enterprise Kubernetes container platform. For more information on CRI-O check out the CRI-O community web site and blog.

Crictl

The Crictl utility is a tool for testing Kubernetes Container Runtime Interface (CRI) compliant daemons.  The Crictl utility communicates using the CRI protocol to any daemon that provides the CRI interface. Currently CRI-O and containerd provide this. Traditionally, Crictl has been targeted for developer use cases, namely testing, validation, and debugging of container runtimes. But, the Crictl can also be a great tool for administrators trying to diagnose issues in your Kubernetes/CRI-O configuration.  Crictl always reports what Kubernetes sees.

Podman

Podman is a tool designed for managing pods and containers without requiring a container daemon.  Pods and containers processes are created as children of the Podman tool. Podman does NOT speak CRI. It does not communicate directly with CRI-O.  However Podman, like Buildah, shares the same backend datastores as CRI-O.

Podman can do a lot of things that Crictl can not.

• restarting stopped containers. CRI does not include a function for restarting a container once it’s stopped, so Crictl and CRI-O don't do it.
• Management and creation of container images
  • Push, commit, configure, build

Podman’s CLI is based on the Docker CLI giving users a familiar CLI experience.  It is intended to be a user friendly interface and is capable of providing summaries of containers, images, and their actions.  Podman manages pods, but can work with containers outside of pods as well. Crictl does not do this, since kubernetes has no concept of containers outside of pods.

Podman is intended to be used without requiring a daemon.  Podman unlike Crictl does not require a running CRI-O daemon.  If CRI-O for some reason is not responding, Podman can still examine the state of containers and images on your system.
Note: Currently Podman and CRI-O do NOT share the same library for identifying containers, yet.  This means, Podman cannot list containers created by CRI-O and CRI-O/Crictl does not know about containers created by Podman.  We plan on fixing this in the future when we merge libpod (Podman’s container management library) into CRI-O. We are just now investigating this, but are concerned about potential performance issues when running hundreds or thousands of containers. With Podman running a few containers there are no performance issues.

Let’s examine a few questions I have heard:

Question: The Docker CLI connects to the Docker Daemon to make requests to build and run containers - does Podman work the same way with CRI-O?

Not exactly.  Both Podman and the Docker CLI provide a developer friendly CLI to work with containers on the command line. They both implement simple commands to build and run containers which are almost identical. Podman can also consume Dockerfiles. This makes it easy for a user to switch from the Docker CLI to Podman without much thought and without needing to change their scripts.

docker run -> podman run

docker build -> podman build

But, unlike the Docker CLI, Podman does not rely on a Container Engine such as the Docker daemon or CRI-O.  Podman creates the containerized processes and makes the necessary changes on disk itself. Podman relies on a library called containers/images for pulling Container Images from Registry Servers. It also uses containers/storage to manage images on disk on the Container Host. These are the same libraries that CRI-O uses, so container images pulled by Podman can be shared and used with CRI-O.

Question: The Crictl utility connects to the CRI Daemon to make requests to list containers, etc - does Podman work the same way with CRI?

No, Podman does not talk directly to the CRI daemon. As stated earlier, Podman implements the building and running of containers outside the scope of the CRI interface definition. Podman does not communicate with using the CRI protocol. Instead, Podman creates containers using runc, and manages storage using containers/storage. Technically, Podman launches conmon which launches and monitors the OCI Runtime (runc).  Podman can exit and later reconnect to conmon to talk to the container.  Runc stops running once the container starts.

Question: I’m confused, I thought Podman can create Pods? Doesn’t this mean that it somehow talks to Kubernetes?

A Pod by definition, is a group of one or more containers, usually sharing several kernel namespaces (network, IPC, PID).  libpod has a looser definition of a pod, in that it is just a group of containers that you can manage together, for example start, stop, inspect.  Libpod pods can be used to implement Pod constructs which are very similar to the rules used by Kubernetes or groups of containers which have their own rules, for example, share network, but not IPC, etc. While Kubernetes popularized this concept, it is not necessary to run Kubernetes to run a group of containers in a pod. Kubernetes provides much more than just groups of containers - it offers scheduling across large clusters, and a standard json/yaml language for defining sophisticated sets of resources/objects such as Persistent Volumes, Replication Controllers, etc.

As far as the Podman name goes, the idea is to have a tool that can manage individual containers AND pods. Podman relies on an underlying library called libpod which provides a lot of the functionality. While kubernetes invented the term pod, the greater containers community has expressed interest in working with pods outside of Kubernetes.  We would like to experiment with these concepts such as defining what happens when one container exits in a multi-container pod, should the pod exit, or should the pod restart the container.  Podman and libpod provide a versatile, but simple interface for running containers outside the scope of container orchestration.

Question: Is Crictl strictly a tool to test the Kubernetes CRI interface?

Yes, this has been the major focus of Crictl up till this point, but some maintainers of Crictl have begun to push it as an alternative to the Docker CLI for launching pods/containers, although it uses a very different set of sub-commands than the Docker CLI or Podman.  We believe that Podman is an easier solution for people who just want to run containers, since it does not require you to run a Container Engine (daemon), and it follows the CLI which people are familiar with.

Also, the Kubernetes Container Runtime Interface (CRI) is an imperative, container-centric interface. The CRI interface relies on the Kublet to ensure the actual state is driven toward the defined state. The CRI interface does not support all commands necessary for managing containers in Pods. As such, Crictl can only execute commands that are defined in CRI.  As explained above, some key container management features like restarting existing containers has not been defined in CRI.

Question: Can Crictl be used to communicate with CRI-O, Containerd and Docker?

The CRI interface can be thought of a subset of the features necessary in any given Container Engine. This subset is exactly what Kubernetes needs to manage the actual state. A container engine can implement many features which are outside the scope of the CRI interface.

In short, developers would typically use Podman or the Docker CLI to work with containers directly. Kubernetes administrators would find Crictl more useful to debug Kube CRI interface itself.

Question:  If a developer prefers to use the Docker CLI, docker build or other Docker Developer Tools, can they still take their images and run them in CRI-O?

Yes. The reason is, docker build produces either an OCI image or a Docker image. Down the road a developer can seamlessly switch to Buildah, instead of docker build, without worry. Either way, the output is an OCI compliant image which can be run with Docker, Podman, or CRI-O.

Docker, Podman, and CRI-O are all designed to run OCI compliant Container Images. CRI-O can pull and run any container image that is stored in a container registry whether it was built using docker build, buildah or any of the myriad of new, open source tools springing up to build container images.

The OCI (Open Container Initiative) has given us a standard definition for what a container image is, a standard way to run containers, and a reference implementation of the Container Runtime (runc).  This means Podman, Docker, CRI-O, Buildah, and Skopeo can all interact with standard images and standard runtimes.

Bottom line

The bottom line is, Podman is fully independent of CRI-O but can manage CRI-O’s storage back end. Eventually it will be able to work fully with CRI-O data and be able to manage the content behind the scenes.  One nice thing about using Podman for diagnosing issues with CRI-O is that it can work even if an issue occurs with the kubelet or CRI-O. The Crictl utility can help you diagnose and understand the CRI protocol, as long as CRI-O is running.

Both tools are great additions to an administrator’s arsenal of container tools.

Crictl checks the front entrance, while Podman examines the foundation.  :^)