Kubernetes architecture: How to use hierarchical namespaces for multiple tenants

Note that this article is scoped to discuss upstream Kubernetes. As of March 1, 2022, hierarchical namespaces are not supported in OpenShift; you can see the latest updates on hierarchical namespaces support in this Knowledgebase article.

A multitenant Kubernetes architecture is one in which a single Kubernetes cluster hosts several independent organizational units. Each tenant is isolated and autonomous within the boundaries of its security constraints. You might use a multitenant cluster with your company's internal teams and workgroups as well as with your external customers. For example, a commercial service that provides instances of its business application to distinct customers is a good example of a multitenant application, as illustrated in the figure below.

Using a multitenant architecture is more cost-effective than spinning up a new cluster for each organizational unit. It's a dollars-and-cents benefit. But, along with that cost advantage comes a certain degree of management complexity, particularly around Kubernetes namespaces.

Kubernetes namespaces are central to implementing a multitenant architecture. When multitenant architectures first appeared on the scene, all the namespaces that defined the various tenants in a cluster were peers to each other. Each namespace was fully isolated from all others. The administrative overhead that peer namespaces incurred made managing hierarchical relationships among tenants difficult.

In 2020, Kubernetes introduced the hierarchical namespace (HNS) framework. HNS helps you manage the security and capabilities of tenants with less effort than the flat, peer-to-peer model. Under HNS, administrators can organize tenants according to a group hierarchy and allocate capabilities accordingly. 

Note that if you're using Red Hat OpenShift, hierarchical namespaces are not supported in OpenShift as of March 1, 2022; you can consult Red Hat's Products and Services Knowledgebase for information about this feature's support.

This article covers the basic concepts and techniques for implementing a multitenant architecture under Kubernetes. First, I'll cover core concepts about Kubernetes namespaces, and then I'll demonstrate how to implement hierarchical namespaces in a multitenant cluster. I'll also provide a bonus section that has both a Bash script you can use to create an example hierarchical namespace as well as an interactive set of lessons that show you how to install, create, and use hierarchical namespaces in a Kubernetes cluster.

I'll start with the nature of namespaces.

Kubernetes namespaces, roles, and capabilities

Namespaces are an abstract resource that enables Kubernetes to set boundaries for other resources. For example, once I create a namespace, I can assign Kubernetes resources such as services, pods, secrets, and config maps to that namespace. Then I can restrict access to that namespace and the capabilities that can be executed within the namespace by binding the namespace to a Kubernetes role.

[ Learn about the Zero-Trust approach to designing security architectures. ]

For example, the statement below is a command executed using the kubectl command-line-interface (CLI) tool that creates a role named biz-app-admin. The role biz-app-admin is dedicated to the namespace biz-app. The biz-app-admin role has permissions to get, list, watch, create, update, and delete pods within the namespace biz-app:

$ sudo kubectl -n biz-app create role biz-app-admin \
   --verb=get,list,watch,create,update,delete --resource=pod

On the other hand, the following statement creates a role named biz-app-sre that is also dedicated to the namespace biz-app. However, the role biz-app-sre only has permissions to update existing pods in the Kubernetes cluster:

$ sudo kubectl -n biz-app create role biz-app-sre \
    --verb=update --resource=pod

Thus, while biz-app-admin has full administrative permissions for pods in the namespace biz-app, biz-app-sre can only update pods.

Kubernetes namespaces and roles provide system administrators with the control required to support multiple tenants using the same application in a single Kubernetes cluster. However, it all comes with a tradeoff. Namespace and role management can become quite complicated when you're dealing with a complex application with many tenants. Not only will each tenant get its own namespace, but subordinate organizations within the tenant can also get namespaces. So you can end up with a set of namespaces that look like this:

biz-app-company-one-finance
biz-app-company-one-hr
biz-app-company-one-management
biz-app-company-two-finance
biz-app-company-two-hr
biz-app-company-two-management

When you have a tenant with many namespaces, you run into situations where redundant role assignment can occur. For example, imagine you have three namespaces that are each associated with two cluster roles:

–– Namespace: biz-app-company-one-finance ––
Cluster role: biz-app-company-one-admin
Cluster role: biz-app-company-one-sre

–– Namespace: biz-app-company-one-hr –– 
Cluster role: biz-app-company-one-admin
Cluster role: biz-app-company-one-sre

–– Namespace: biz-app-company-one-management –– 
Cluster role: biz-app-company-one-admin
Cluster role: biz-app-company-one-sre

As you can see, all three namespaces (biz-app-company-one-finance, biz-app-company-one-hr, and biz-app-company-one-management) are all associated with the cluster roles biz-app-company-one-admin and biz-app-company-one-sre. This has the potential to become a maintenance nightmare. For example, imagine that the namespace biz-app-company-one-hr required that the role biz-app-company-one-admin have a special permission. Adding that new permission to the role biz-app-company-one-admin will make the added permission cascade to the other namespaces as well.

[ Learn 16 steps for building production-ready Kubernetes clusters. ]

To avoid this hazard, the system admin will need to create a new role that has the existing permissions of biz-app-company-one-admin and the new, required permission. Then the system admin must detach the role biz-app-company-one-admin from the namespace biz-app-company-one-hr and attach the newly created role to that namespace. And, this all has to be done while the application is running!

Clearly, a better way is needed. What if you could turn this set of namespaces:

biz-app-company-one-finance
biz-app-company-one-hr
biz-app-company-one-management
biz-app-company-two-finance
biz-app-company-two-hr
biz-app-company-two-management

… into a hierarchy that looks like this?

biz-app
├── company-one
│ ├── company-one-finance
│ ├── company-one-fs
│ └── company-one-management
├── company-two
│ ├── company-two-finance
│ ├── company-two-fs
│ └── company-two-management

Fortunately, by using hierarchical namespaces, you can.

Using hierarchical namespaces enables an application to set a role(s) with a base set of permissions that cascades down to subordinate namespaces. Then, if the subordinate namespace needs a special set of permissions, system admins can create that particular role and assign it to the subordinate namespace.

Hierarchical namespaces provide an efficient way to manage multitenant applications. Here are the details about how they work.

Implement a hierarchical namespace architecture under Kubernetes

Kubernetes does not ship with hierarchical namespaces out of the box. You have to install two components. One is the Hierarchical Namespace Controller (HNC), which is installed within the control plane of the Kubernetes cluster. The other is the HNS kubectl CLI tool plugin. The plugin allows you to create hierarchical namespaces using the command set kubectl hns from a terminal's command line.

Install HNC and hnc plugin

To get hierarchical namespaces up and running, you need to install the HNC and the kubectl plugin. You can use the following Bash script to install both items:

#!/bin/bash

# Declare the script variables for the version and platform of the HNS installation
HNC_VERSION=v0.8.0
HNC_PLATFORM=linux_amd64

# Twiddle with the existing namespaces to exclude for HNS operations
kubectl label ns kube-system hnc.x-k8s.io/excluded-namespace=true --overwrite
kubectl label ns kube-public hnc.x-k8s.io/excluded-namespace=true --overwrite
kubectl label ns kube-node-lease hnc.x-k8s.io/excluded-namespace=true --overwrite

#Install the Hierarchical Namespace Controller
kubectl apply -f https://github.com/kubernetes-sigs/multi-tenancy/releases/download/hnc-$HNC_VERSION/hnc-manager.yaml

#Put the installation to sleep for 60 seconds so that Kubernetes can adjust itself
date +"%s"
sleep 60
date +"%s"

# Go a directory that is in the system $PATH
cd /usr/local/bin

# Install the HNS kubectl plugin in that directory
curl -L https://github.com/kubernetes-sigs/multi-tenancy/releases/download/hnc-$HNC_VERSION/kubectl-hns_$HNC_PLATFORM -o ./kubectl-hns

# Give the plugin executable permission
chmod +x ./kubectl-hns

# Do a test execute on the plugin installation
kubectl hns

Copy this into a script file and give it any name you want (for example, setup.sh). Then execute it.

After the installation completes, you'll see the command line help documentation displayed in the terminal window.

The next step is to create some hierarchical namespaces.

Create hierarchical namespaces

Once the HNC and the kubectl plugin are installed, you can create hierarchical namespaces. You can create a hierarchical namespace for Kubernetes imperatively at the command line with the syntax:

kubectl hns create <CHILD_NAMESPACE> -n <PARENT_NAMESPACE>

Where:

<CHILD_NAMESPACE> is the name of the subordinate namespace you want to create.

<PARENT_NAMESPACE> is the name of the parent namespace to which the child namespace will be subordinate.

For example, to create a subordinate namespace named company-one for an existing namespace named biz-app, execute:

$ sudo kubectl hns create company-one -n biz-app

This command will create a hierarchical structure. Also, you can make an existing namespace subordinate to another existing namespace with:

$ sudo kubectl hns set <NAMESPACE_B> --parent <NAMESPACE_A>

Where:

<NAMESPACE_B> is an existing namespace.

<NAMESPACE_A> is an existing namespace.

Be advised that to create a hierarchical structure using existing namespaces, the entity executing the command set needs administrative permissions for the entire cluster.

Once you have established the hierarchical namespace structure, add roles to the namespace that constrain the capabilities of Kubernetes resources operating under the designated namespace. Here's how that works.

Apply cascading capabilities to a hierarchical namespace

The advantage that hierarchical namespaces bring to architectural design is that they enable architects and admins to build cascading permission structures to a Kubernetes cluster.

Imagine that you have a hierarchical namespace structure such as this one, in which an application named biz-app supports two tenant companies:

biz-app
├── company-one
│ ├── company-one-finance
│ ├── company-one-hr
│ └── company-one-management
├── company-two
│ ├── company-two-finance
│ ├── company-two-hr
│ └── company-one-management

Now imagine that you want to make it so that the application administrators have full administrative capability over all pods in the namespace biz-app, including both the parent and the subordinate namespaces. To do this, create a Kubernetes role named biz-app-admin and give that role complete control over all pods in the namespace biz-app. This is an example of how to create the biz-app-admin Kubernetes role imperatively:

$ sudo kubectl -n biz-app create role biz-app-admin --verb=* --resource=pod

Not only does this command apply the role to the top-level namespace, biz-app, but the role also cascades down to all subordinate namespaces.

Create more roles

Next, imagine that you want to create two other Kubernetes roles, company-one-sre and company-two-sre. These roles will only have the permission to update existing pods in their respective namespaces.

Create the role company-one-sre with update-only permission:

$ sudo kubectl -n company-one create role company-one-sre --verb=update --resource=pod

Notice that the role company-one-sre is assigned to the namespace company-one using the option -n.

Create the role company-two-sre with update-only permission:

$ sudo kubectl -n company-two create role company-two-sre --verb=update --resource=pod

The role company-two-sre is assigned to the namespace company-two using the option -n.

This figure shows the hierarchical structure resulting from the various role assignments.

Now, I'll do some analysis.

Verify Kubernetes namespace hierarchy

In the previous section, you created three roles named biz-app-admin, company-one-sre, and company-two-sre. The role biz-app-admin has complete control over all pods in the biz-app namespace as well as the namespaces that are subordinate to biz-app. On the other hand, the namespaces company-one-sre and company-two-sre only have permission to update existing pods in their respective namespaces. But, they can also update pods in their subordinate namespaces. For example, company-one-sre can update pods in the namespace company-one and company-one-finance.

Now that you've set up the hierarchical permission structure according to various Kubernetes roles, verify that these roles cascade into subordinate namespaces as expected. Verify that a role is in force by running the command kubectl get role against the given namespace. If the command returns the role according to the namespace, the role is in force.

First, see if you can get the role biz-app-admin from the default namespace. When executing the command, call the default namespace by not providing a namespace option (-n):

$ sudo kubectl get role biz-app-admin
Error from server (NotFound): roles.rbac.authorization.k8s.io "biz-app-admin" not found

As you see, you get an error. This makes sense because the role biz-app-admin is not assigned to the default Kubernetes namespace.

Next, get the role biz-app-admin by declaring biz-app as the namespace option:

$  sudo kubectl get role biz-app-admin -n biz-app
NAME            CREATED AT
biz-app-admin   2021-12-13T20:16:05Z

This call is successful because the role biz-app-admin was assigned initially to the namespace biz-app.

Try to get the role biz-app-admin from a namespace that is subordinate to the namespace biz-app. In this case, go down two levels to company-one-finance:

$  sudo kubectl get role biz-app-admin -n company-one-finance
NAME            CREATED AT
biz-app-admin   2021-12-13T20:16:05Z

Once again, the call was successful.

This time, try to get the role biz-app-admin from the other company namespace, company-two. Remember, company-two is subordinate to the namespace biz-app. Once again, go down two levels:

$  sudo kubectl get role biz-app-admin -n company-two-hr
NAME            CREATED AT
biz-app-admin   2021-12-13T20:16:06Z

Again, success! Why? Because the role biz-app-admin was assigned to the namespace at the highest level of the hierarchy, biz-app. The namespace biz-app-admin can be retrieved from anywhere.

Move on to the other Kubernetes roles. Look at the role company-one-sre, which was created against the namespace company-one. First, try to get the role from the company-one namespace:

$ sudo kubectl get role company-one-sre -n company-one
NAME              CREATED AT
company-one-sre   2021-12-13T20:16:19Z

Success!

[ Learn more about Kubernetes' architecture. Download the eBook Kubernetes patterns: Reusable elements for designing cloud-native applications. ]

Now try to get the role from the namespace biz-app. As mentioned, the namespace biz-app is the root of the namespace hierarchy:

$ sudo kubectl get role company-one-sre -n biz-app
Error from server (NotFound): roles.rbac.authorization.k8s.io "company-one-sre" not found

This call fails because the role company-one-sre is unknown to the root namespace biz-app. Remember: the role company-one-sre was initialized against the namespace company-one. The root namespace biz-app is superior to the namespace company-one. Hence, the namespace company-one-sre is unknown.

Look at the namespace company-one-finance, which is subordinate to the namespace company-one. See if you can get the Kubernetes role company-one-sre from the namespace company-one-finance:

$ sudo kubectl get role company-one-sre -n company-one-finance
NAME              CREATED AT
company-one-sre   2021-12-13T20:16:19Z

Success again!

Finally, see if you can get the role company-one-sre from the namespace company-two:

$ sudo kubectl get role company-one-sre -n company-two
Error from server (NotFound): roles.rbac.authorization.k8s.io "company-one-sre" not found

Failure! This is to be expected because company-one-sre has no association at all with the namespace company-two or any of the namespaces that are subordinate to company-two.

These exercises should give you a clear idea of how hierarchical namespaces work and how cascading permissions can be applied to a hierarchical namespace using Kubernetes roles. Of course, to make the scenario entirely operational, you'd have to create a set of Kubernetes rolebindings that associate a particular user, group, or process with a role. This is work to be done, but once the hierarchical namespaces and corresponding roles are created, the pieces are in place to make that final piece of configuration possible.

Putting it all together

If you're in an IT department supporting multiple customers using a company's enterprise-level web application, multitenant clusters can be more cost-effective than running each customer in its own Kubernetes cluster. The tradeoff is that a multitenant cluster can get unwieldy when using out-of-the-box Kubernetes namespaces to isolate tenants. Hierarchical namespaces make it all a lot easier.

You'll have to install an HNC on every cluster where you plan to run hierarchical namespaces. Also, to get the kubectl CLI tool to support working with hierarchical namespaces, you'll have to install a special plugin on the machine you're using to administer your Kubernetes clusters. Fortunately, these tasks are not difficult to do. As this article demonstrates, it's nothing more than executing two sets of commands. You also have to alter your thinking to adjust to working with roles that cascade permissions over portions of the namespace hierarchy. It's a bit of a change, but it's not rocket science.

Hierarchical namespaces add a new dimension to managing a multitenant Kubernetes cluster. Acquiring mastery working with them will take some time, but it's time well spent given the benefits you get in return.

Bonus sections:

Get hands-on with hierarchical namespaces

I've created this set of hands-on lessons that you can use to try out the details of installing, creating and using hierarchical namespaces in Kubernetes.

Create demonstration namespaces and roles

You can use the Bash script below to create the hierarchical namespaces and roles used as examples in this article. Once you have a Kubernetes cluster configured to run hierarchical namespaces, you can copy the contents of the listing into a Bash file you create on the machine where you're running kubectl. Execute that Bash file to create the biz-app hierarchical namespaces and roles used in this article.

#!/bin/bash

# Create the biz-app namespace
kubectl create namespace biz-app

# Create the company-one namespace …
kubectl hns create company-one -n biz-app
# … and its subordinate namespaces
kubectl hns create company-one-finance -n company-one
kubectl hns create company-one-hr -n company-one
kubectl hns create company-one-management -n company-one

# Create the company-two namespace …
kubectl hns create company-two -n biz-app
# … and its subordinate namespaces
kubectl hns create company-two-finance -n company-two
kubectl hns create company-two-hr -n company-two
kubectl hns create company-two-management -n company-two

# List the namespaces for biz-app
kubectl hns tree biz-app

# Create the roles
kubectl -n biz-app create role biz-app-admin --verb=* --resource=pod

kubectl -n company-one create role company-one-sre --verb=update --resource=pod

kubectl -n company-two create role company-two-sre --verb=update --resource=pod