Nested OpenShift using OpenShift Virtualization

As I hinted in my previous posts, OpenShift Virtualization opens up new infrastructure design possibilities. I believe one of the biggest things that will come down the pipeline is OpenShift on OpenShift. That's right I'm talking about nested OpenShift thanks to OpenShift Virtualization.

I'm guessing the first question you have is why? Good question, the benefits of nested OpenShift include:

• Ease of deployment to virtual environments. Now you can have benefits of virtual machines without paying extra for the hypervisor. OpenShift Virtualization uses KVM and Libvirt which are tried and true virtualization technologies used since RHEL 6.
• Better resource utilization. OCP control nodes (aka masters) are perfectly happy running in virtual machines so you don't have to worry about dedicating bare metal servers who's resources will be mostly unused.
• With the ease of deploying in virtual environments, you can have a robust GitOps CI/CD pipeline for testing and rolling out OCP software and configuration updates.
• Instead of dealing with large clusters you can deploy more manageable cluster sizes. This also allows you to deploy clusters with various configurations to better serve requirements of the app teams.
• Allows for highly available cluster designs.

Concepts

Before we start lets get familiar with some nomenclature I will be using going forward.

• Infra clusters are Bare Metal OpenShift clusters that will serve as virtualization hosts.
• Tenant clusters are OpenShift clusters hosted partially or fully on Infra clusters.

Architectures

Fully Virtualized OCP cluster

The most basic nested OCP design is a single Infra cluster hosting multiple Tenant clusters. Each tenant cluster occupies a single namespace in the Infra cluster. Due to the rapid pace of development in the Kubernetes world, a GitOps pipeline is absolutely crucial to keep up with the latest changes. It also allows individual engineers to test their code changes without disrupting the work of others. This design is also well suited for Dev and QA environments on the application side.

Partialy Virtualized OCP Cluster

The next design only deviates slightly by hosting the Tenant cluster control nodes on the Infra clusters and using bare metal servers as the workers. This has the same benefits as the previous design with the addition of being able to test bare metal OCP deployment. This design is well suited for Dev,QA, UAT and Pre-Prod testing.

Highly Available Partialy Virtualized OCP Cluster

Now we get to a really interesting architecture. This design requires three Infra clusters. The Tenant cluster control VMs are distributed across three different infra clusters. This is similar to having availability sets in AWS. Just like the previous designs the workers can be bare metal or virtual. The major advantage of this design is in its resiliency. Even if you lose an entire Infra cluster, all the tenant clusters will still be full operational. Once the Infra cluster is recovered, you can easily restore the control nodes for the tenant cluster. This resilient design is well suited for production environments, especially ones that have demanding uptime requirements.

Now that we have our designs and value proposition, the next question you might have is how? In my next article I will guide you through creating a basic nested OCP cluster. Stay tuned!