As an engineering organization, Red Hat is investing in CRI-O and Podman, participating in the Open Containers Initiative standards body, testing performance and security, as well as driving architectural changes in a number of container projects because the underlying shared components help drive innovation in its products like Red Hat OpenShift and Red Hat Enterprise Linux. These investments are closely related to the operating system itself and provide our customers with the best products we can produce.
Engineering is a Zero Sum Game, or is it?
We have often heard that engineering is a zero sum game - if you invest $1 in engineering, you get $1 back in value. Every year, when budgets are crafted, managers have to select the hungry projects that get those investment dollars. By doing so, they are starving 20-plus other important projects from much needed dollars. It’s a tragedy, but it has to be done because "wants" are infinite while "budgets" are limited. Managers have to stack rank engineering investments and make tough choices - or, do they?
For years, standard engineering & supply chain logic dictated that a dollar invested is a dollar returned (at best). Furthermore, whenever we encounter an engineering problem, we have three choices for that dollar - we can build, buy, or partner. With traditional engineering logic this has been the standard calculus for years.
But, with community driven, open source software, there’s another option. Instead of building, buying, or partnering with another company to gain access to a piece of technology, we can invest in an open source community. This has the potential to provide value far beyond solo R&D investment. We can invest $1 in engineering upstream and potentially get back more than a dollar, perhaps $2 or $3 in value because there’s a community of other engineers investing - some individuals, some sponsored by businesses. That changes everything.
Red Hat is an engineering driven organization and we are committed to upstream community projects. These projects serve as the foundation, and sort of a supply chain for our products, completely changing the traditional investment model. Red Hat contributes to many projects in the container space, a plethora of lower level tools and libraries, as well as vendor neutral standards.
These engineering investments often pay dividends worth much more than $1 for $1. Each of these investments strengthen the ecosystem, offering a wide variety of technology to choose from when building open source products. In this model, everybody can win - customers, community, Red Hat and its partners, as well as our competitors.
When building products, not unlike traditional manufacturing, vendors are free to make their own technology commitments, differentiating their products based on customer needs like reliability, scalability, performance, cost, etc. A flourishing ecosystem of full of choice is good for everyone.
Why CRI-O & Podman?
The container engine changed the way the world finds, runs, builds, and shares containers. Since the beginning it has been an important component in any container platform. Red Hat was an early contributor in the Linux container space, because we saw the value it brought to the Linux ecosystem.
Shortly after Linux containers became popular, we realized that the tooling for building and running Linux containers was only part of the picture. We needed to focus on container orchestration as well. We looked to find the best way to develop container tools for single hosts and in clusters, standardizing on Kubernetes.
We heard from customers that small modular tools which allowed for quicker experimentation and development of features like rootless were priorities, and decided that investing in CRI-O and Podman were the best way to meet these needs for customers and the community. Red Hat has a lot of expertise in the lower level components and decided that we could contribute here.
Red Hat decided to invest in Podman because it’s an easy change for anyone used to the Docker command line, but doesn’t require a daemon to be running when one isn’t needed. Also, Podman shares many of the same underlying components with other container engines, including CRI-O, providing a proving ground for new and different container runtimes, and other experiments in underlying technology like CRIU, Udica, Kata Containers, gVisor, etc.
Since the early days of container engines about five years ago, the world has changed to embrace Kubernetes as the de facto container orchestration engine. Today, the container engine is still a standalone piece of technology used by many end users, but it’s also what ties the Kubelet to the Linux kernel completing the container host in the context of Kubernetes and Red Hat OpenShift.
This container stack within Red Hat Enterprise Linux and Red Hat Enterprise Linux CoreOS serves as part of the foundation for OpenShift. As can be seen in the drawing below, the CRI-O stack in OpenShift shares many of its underlying components with Podman. This allows Red Hat engineers to leverage knowledge gained in experiments conducted in Podman for new capabilities in OpenShift.
Red Hat believes that to deliver reliable software to our customers, we need to have expertise with the underlying technology. The way to gain this expertise is by participating at all levels of the stack. Compiling and shipping is not enough. We believe a vendor must actually understand emerging technical and business problems (our users are technical, so those business problems are often technical in nature), and drive the upstream communities to meet to provide solutions in security, performance, and major architectural changes.
Here’s just a few of the upstream projects Red Hat participates in, which have provided insights and knowledge about emerging technical and business problems.
Investing in these technologies and others is strategic because it gives OpenShift the ability to add new capabilities like rootless containers, rootless builds, custom SELinux policies, User Namespaces, etc. No technology is perfect, but participation is part of driving improvement.
Other engineering organizations have committed to similar low level research and development. For example, Google started the V8 project, and some couldn’t understand why. But, the Chrome web browser changed the way we use web apps. People questioned Google’s investment in V8 because it wasn’t directly related to anything they did. However, this investment contributed to the success of the Chrome web browser.
Similarly, Red Hat believes it can lead by developing innovations in the container engine, and runtime space. This may lead to creative new features in these communities as well as Red Hat OpenShift Container Platform. Other communities may also benefit by pivoting off of our ideas. This can lead to better technology for everyone.
At Red Hat, we love tackling difficult engineering problems and you can discern a lot about a technology company by analyzing its research and development - what problems it’s tackling.
With Red Hat, that research and development is done transparently in upstream communities like CRI-O, which we recently contributed to the Cloud Native Computing Foundation (CNCF). Red Hat is investing heavily in container engines and runtimes by participating in the upstream CRI-O and Podman projects. These tools are community driven, open source, and focused on meeting the technical requirements that people want - small, fast, a more security focused architecture, as well as new features like rootless, daemon-less, OCI compliant, etc. Come join our communities (Podman & CRI-O), your participation is welcome!
About the authors
At Red Hat, Scott McCarty is Senior Principal Product Manager for RHEL Server, arguably the largest open source software business in the world. Focus areas include cloud, containers, workload expansion, and automation. Working closely with customers, partners, engineering teams, sales, marketing, other product teams, and even in the community, he combines personal experience with customer and partner feedback to enhance and tailor strategic capabilities in Red Hat Enterprise Linux.
Daniel Walsh has worked in the computer security field for over 30 years. Dan is a Senior Distinguished Engineer at Red Hat. He joined Red Hat in August 2001. Dan leads the Red Hat Container Engineering team since August 2013, but has been working on container technology for several years.