In a distributed bare metal environment, there have traditionally been two options for control plane architecture when dealing with large-scale OpenShift deployments:

  1. Run one control plane at a centralized location, with many remote workers.
  2. Run the control plane on each remote node (Single Node OpenShift).

Hosted Control Planes, based on the upstream HyperShift project, is currently in tech preview for bare metal, offering us a third option:

  1. Run many containerized control planes at a centralized location, each connected to a small number of remote workers.


This approach can offer a few benefits:

  • Reduced resource consumption on the remote nodes compared to Single Node OpenShift.
  • Faster deployment times at the remote sites compared to Single Node OpenShift.
  • A much more scalable architecture when compared to a single control plane with many remote workers. Rather than having one control plane (limited to three nodes) serving hundreds of workers, you can deploy hosted control planes as needed, which run as pods on worker nodes as part of a centralized cluster. The number of worker nodes in this centralized cluster can be scaled as demand dictates.

A previous blog post has shown how to create Hosted Control Plane clusters using the CLI. In a production environment, you need something that allows you to scale your deployments consistently and reliably. This is where the OpenShift GitOps Operator, based on ArgoCD, comes in.

Setting up the management cluster

My management cluster consists of three bare metal servers. The cluster has ODF installed, which provides persistent storage. Persistent storage (meaning an available StorageClass) is required for ACM/MCE, as well as HyperShift (used to store the hosted cluster etcd database).


The management cluster requires either RHACM or the Multicluster Engine Operator. Once either of these operators is installed, we will enable the hypershift-preview feature, as well as CIM (Central Infrastructure Management), to allow zero-touch provisioning of bare metal worker nodes. Using CIM to provision bare metal workers is known as the Agent platform when dealing with Hosted Control Planes.

See this document for instructions on how to enable the hypershift-preview feature. Since ACM 2.7/MCE 2.2, the HyperShift Operator is automatically installed on the local-cluster after it is enabled.

See this document for instructions on how to enable Central Infrastructure Management (the Assisted Service). This feature will allow the management cluster to provision bare metal worker nodes by automatically mounting the installer ISO and installing OpenShift.


Since the control plane for our new cluster will be running as pods on the management cluster, we will use MetalLB to expose the API endpoint for the new cluster.

See this document for instructions on how to install the MetalLB Operator.


See this document for instructions on how to install the GitOps Operator.

We will provide an install-config.yaml file as the values for a Helm chart that will deploy a hosted cluster on top of our management cluster. The GitOps Operator will apply the Helm chart for us and ensure that the specified configuration stays in sync with what is running on the management cluster.


The hosted cluster still needs DNS entries for api.<cluster-name>.<domain> and *.apps.<cluster-name>.<domain>.

Since we will be serving the API using MetalLB (layer 2) on the management cluster, the address we choose for api.<cluster-name>.<domain> needs to be in the same subnet as the management cluster.

*.apps.<cluster-name>.<domain> will be served using MetalLB running on the hosted cluster workers, therefore, that IP address needs to be in the same subnet as the hosted cluster workers.

Creating an Install Config

We can use a regular install-config.yaml file as the input to our Helm chart, with just a few additional parameters. Since the control plane will be hosted on the management cluster, we only need to specify worker nodes in our install config.

The install-config.yaml file needs to be stored in a Git repository. Since this file contains sensitive information, we'll keep it in a private repository that requires authentication to access.

apiVersion: v1

# Additional parameters for hypershift-helm
clusterImageSet: # Required

baseDomain: <cluster_domain>
- name: worker
replicas: 1
name: example-cluster-name
- cidr:
hostPrefix: 23
networkType: OVNKubernetes
apiVIP: <api_address> # Should be in the same subnet as the management cluster
ingressVIP: <ingress_address> # Should be in the same subnet as the hosted cluster worker nodes
- name: openshift-worker-0
role: worker
address: "redfish-virtualmedia://<bmc_ip_address>/redfish/v1/Systems/1"
username: <username>
password: <password>
bootMACAddress: <nic1_mac_address>
hctl: "1:0:0:0"
- name: eno1
type: ethernet
macAddress: <nic1_mac_address>
state: up
enabled: true
dhcp: true
enabled: false
pullSecret: "<pull secret>"
sshKey: |
ssh-rsa ...


Creating the ArgoCD app

Since the release of OpenShift GitOps 1.8, the operator has supported the ability to define multiple sources for an application. This is important for our Helm chart, since your Values file (install-config.yaml) will be in a different repository than the Helm chart itself.

Since the Git repository that hosts our install-config.yaml requires authentication, we need to configure it with credentials in ArgoCD first. See the ArgoCD docs for instructions on adding a private repository.

Once we have created our install-config.yaml and committed it to our private repository, we can create an ArgoCD app:

cat << EOF | oc apply -f -
kind: Application
name: hypershift-cluster
namespace: openshift-gitops
server: https://kubernetes.default.svc
project: default
- repoURL: ''
chart: deploy-cluster
targetRevision: 0.1.12
- $values/helm/install-config-hypershift.yaml
- repoURL: ''
targetRevision: main
ref: values

In the example above, we are pulling the Helm chart from a publicly available repository,, and we are pulling the install-config.yaml file from a private repository, This Helm chart is not supported by Red Hat, it is just a personal project of mine.

The Helm chart is essentially a collection of YAML templates that use the install-config.yaml file as an input to create the required resources on the management cluster (BareMetalHosts, HostedCluster, NodePool, etc).


Once we have created the ArgoCD app, we can log in to the Web UI to look at the status:


The status will show "OutOfSync". Once we click "Sync", ArgoCD will go to work applying the Helm chart on our cluster.

The Helm chart will:

  • Create a new namespace for our cluster.
  • Create BareMetalHost, NMStateConfig, and InfraEnv objects, which will be used by ACM/MCE to provision our bare metal hosts and install OpenShift.
  • Create HostedCluster and NodePool objects. These objects define the configuration for our new hosted cluster.
  • Configure MetalLB on the management cluster to provide access to the hosted cluster API endpoint.
  • Scale the NodePool, which will allow the BareMetalHosts to be attached to the new cluster (Agent platform).
  • Install and configure MetalLB on the hosted cluster workers, to serve the Ingress endpoint.

After about 25-30 minutes, if we check the status of our app again, we will see:


We can check the status of our new hosted cluster via the CLI as well:

[loganmc10@fedoralogan ~]$ oc get hostedcluster,nodepool -n hyper
NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE 4.12.2 hyper-admin-kubeconfig Completed True False The hosted control plane is available


Accessing the cluster

To access the new hosted cluster, we need to get the kubeconfig:

oc get secret -n <cluster-name> <cluster-name>-admin-kubeconfig -o jsonpath='{.data.kubeconfig}' | base64 -d > ~/hosted-kubeconfig

We can then check the status of our new cluster:

[loganmc10@fedoralogan ~]$ export KUBECONFIG=~/hosted-kubeconfig 
[loganmc10@fedoralogan ~]$ oc get co
console 4.12.2 True False False 97s
csi-snapshot-controller 4.12.2 True False False 24m
dns 4.12.2 True False False 2m24s
image-registry 4.12.2 True False False 2m19s
ingress 4.12.2 True False False 23m
insights 4.12.2 True False False 3m3s
kube-apiserver 4.12.2 True False False 24m
kube-controller-manager 4.12.2 True False False 24m
kube-scheduler 4.12.2 True False False 24m
kube-storage-version-migrator 4.12.2 True False False 2m33s
monitoring 4.12.2 True False False 36s
network 4.12.2 True False False 3m14s
node-tuning 4.12.2 True False False 5m42s
openshift-apiserver 4.12.2 True False False 24m
openshift-controller-manager 4.12.2 True False False 24m
openshift-samples 4.12.2 True False False 2m
operator-lifecycle-manager 4.12.2 True False False 23m
operator-lifecycle-manager-catalog 4.12.2 True False False 23m
operator-lifecycle-manager-packageserver 4.12.2 True False False 24m
service-ca 4.12.2 True False False 3m1s
storage 4.12.2 True False False 24m
[loganmc10@fedoralogan ~]$ oc get node
openshift-worker-0 Ready worker 9m10s v1.25.4+a34b9e9


Managing the cluster

In the future, if we want to update the cluster to a newer version, all we need to do is update our original install-config.yaml.



Once we update the Git repository that stores the install-config.yaml file, ArgoCD will notice the change and mark the app as "OutOfSync". Re-syncing the app will cause ArgoCD to update the HostedCluster and NodePool resources, triggering an update to our hosted cluster. This approach will allow us to manage the lifecycle of the cluster the GitOps way.