Deploying multicluster Kubernetes applications with GitOps: What architects need to know
Greater adoption of Kubernetes naturally leads to more multicluster Kubernetes applications, and architects must know how to address the challenges that come with these environments.
One of these is how to standardize and centralize essential functions across all environments, including:
- Application lifecycle management
- Multicluster lifecycle management
This article demonstrates how Red Hat Advanced Cluster Management (ACM) for Kubernetes addresses this challenge by showing an example application deployment. It also shows the two possible ways (at the time of writing) to deploy GitOps-based applications using ACM:
- Managing apps with Git repositories: Deploying resources from Git repositories using the ACM web console's built-in mechanism
- Configuring managed clusters for OpenShift GitOps operator and Argo CD: Registering a set of one or more ACM-managed clusters to an instance of Argo CD before deploying the application to those clusters
The solution relies on ACM for Kubernetes 2.5 and OpenShift GitOps 1.5.4 (or Argo CD).
In addition, you need OpenShift CLI version 4.3.0 or greater for Kustomize.
Finally, you can take advantage of the Web Terminal Operator if you prefer to use only a web browser for the entire deployment.
[ Learn how to build a flexible foundation for your organization. Download An architect's guide to multicloud infrastructure. ]
The following prerequisites must be in place to follow along with the steps in this article:
- Install Argo CD or the Red Hat OpenShift GitOps operator on your Red Hat ACM for Kubernetes platform. I've included a policy to install the OpenShift Pipelines operator in my repo for this article.
- Import one or more managed clusters.
- You'll need the ACM subscription administrator role to leverage all the automation components from the policies. (You need it even if you hold the
The following image shows how ACM implements GitOps with the application subscription model.
The GitOps example in this tutorial is highlighted in the diagram above.
[ Download The path to GitOps to put the right tools, workflows, and structures in place to enable a complete GitOps workflow. ]
subscription.apps.open-cluster-management.io) allow clusters to subscribe to the following source repositories (channels):
- Git repository
- Helm release registry
- Object storage repository
Subscriptions use channels to identify new or updated resource templates. The subscription operator can download templates directly from the storage location and deploy them to targeted managed clusters without checking the hub cluster first. With a subscription, the subscription operator can monitor the channel for new or updated resources instead of using the cluster with the ACM Hub.
Deploy an application using the ACM web console
The administrator doing the deployment must have an access level that can create applications. You can only perform actions that are assigned to your role. The
cluster-admin role can deploy the application, but if you need more granular access, review ACM's role-based access control documentation.
1. From the ACM web console, select Applications → Create application → Subscription.
Select YAML: On to view the YAML in the console as you create your application (or you can import the YAML directly instead of using the interactive web form).
2. The application in this example is called pacman. Enter the following values in the specified fields:
- Name: pacman-app
- Namespace: pacman
Select a namespace from the list. If you have the correct access role, you can also create a namespace using a valid Kubernetes name.
3. Choose Git from the list of repositories.
If you want to learn more about the different options and how they work, refer to the Managing apps with Git repositories documentation.
[ For more on OpenShift, Argo CD, and Tekton, download the complimentary eBook Getting GitOps. ]
4. Click Save, and ACM redirects you to the app dashboard, where you can see all the objects that compose the app and if they are correctly deployed.
Deploy Argo CD ApplicationSets in ACM
An ApplicationSet is an Argo CD subproject that adds multicluster support for Argo CD applications. You can create an ApplicationSet from the product console editor, as shown below.
[ A complimentary guide from Red Hat: The automation architect's handbook. ]
Configure managed clusters for Argo CD
There are two approaches to configuring managed clusters for Argo CD:
- Follow the step-by-step procedure in the official Configuring managed clusters for OpenShift GitOps operator and Argo CD documentation.
- Install the policies in my repo for this article.
NOTE: Remember to grant your user the subscription administrator role before installing the policies.
If you prefer to continue with the automated approach, once the local-cluster complies with the policies, add the clusters you want to the all-openshift-clusters
ClusterSet created by the policy-openshift-gitops-acm-integration policy.
You can choose any name for the
ClusterSet, as ACM supports integration with non-OpenShift clusters. I used all-openshift-clusters in this case because the
ClusterSet consists of OpenShift clusters alone. This is for simplicity, since the
labelSelector of the
Placement object, in charge of integrating the clusters in
ArgoCD, is based on the label with the key/value
vendor=OpenShift, which is the default label with any OpenShift cluster.
1. Select Clusters → Cluster sets → Manage resource assignments.
2. In the next screen, select the clusters, click Review, and then Save.
You can confirm the assignment in two ways. The first method is by using the command interface:
$ argocd cluster list
[ Learn more about cloud-native development in the eBook Kubernetes Patterns: Reusable elements for designing cloud-native applications. ]
The other approach is by accessing the Argo CD web console at Settings → Clusters:
The two managed clusters (highlighted in the screenshot) have been added to the list of clusters managed by Argo CD.
Deploy Argo CD ApplicationSets from ACM web console
Once you have enabled integration between OpenShift GitOps and ACM, you can deploy ApplicationSets with the ACM web console.
Start by deploying the ApplicationSet:
1. From the ACM web console, choose Applications → Create application → Argo CD ApplicationSet.
2. Navigate through the different steps on the web console's interactive form and fill in the relevant data. Here is an example:
Note that the destination namespace could be any value. It's not necessary for the ApplicationSet itself (but mandatory for the ACM form). It is better to leave the value
bgdk to avoid misinterpretation because
bgdk is the namespace that will generate the app deployment itself.
3. Clicking Create, and you are presented with the application. To check it, go to the Applications menu:
Drill down to see its evolution and whether all the elements are correctly deployed in both clusters by clicking on its name.
In this case, select the Route object and check that it is deployed in both clusters and that you have direct access to the URL or can edit the object's YAML.
You can do the same with the Argo CD web console:
1. Check the general status.
2. Drill down by clicking on one of the instances (one per cluster).
Deploy with a YAML file
You can do the same as in the previous section using either the web console or the command line.
Create a YAML file with these contents:
Here is the command-line example:
$ oc apply -f rhacm/deploy/acm-appset.yaml
Other methods include using the OpenShift web console (+Add → Import YAML), Argo CD web console (+NEW APP → EDIT AS YAML), or ACM web console (with the Create application menu).
At the beginning of this article, I outlined the challenge of deploying multicluster applications (whether in private, public cloud, or hybrid environments) while standardizing and centralizing critical functions across all environments. This includes:
- Application lifecycle management
- Multicluster lifecycle management
I have given more attention to the application lifecycle and governance part (thanks to ACM policies) in this article. For insight on multicluster management and observability, please consult the official documentation on these topics:
Navigate the shifting technology landscape. Read An architect's guide to multicloud infrastructure.