Skip to main content

Deploying multicluster Kubernetes applications with GitOps: What architects need to know

Learn how to standardize and centralize critical functions in multicluster applications across hybrid multicloud environments.
Close up of architectural detail

Photo by Scott Webb from Pexels

Greater adoption of Kubernetes naturally leads to more multicluster Kubernetes applications, and architects must know how to address the challenges that come with these environments.

One of these is how to standardize and centralize essential functions across all environments, including:

  • Application lifecycle management
  • Governance
  • Observability
  • Multicluster lifecycle management

This article demonstrates how Red Hat Advanced Cluster Management (ACM) for Kubernetes addresses this challenge by showing an example application deployment. It also shows the two possible ways (at the time of writing) to deploy GitOps-based applications using ACM:

  1. Managing apps with Git repositories: Deploying resources from Git repositories using the ACM web console's built-in mechanism
  2. Configuring managed clusters for OpenShift GitOps operator and Argo CD: Registering a set of one or more ACM-managed clusters to an instance of Argo CD before deploying the application to those clusters

The solution relies on ACM for Kubernetes 2.5 and OpenShift GitOps 1.5.4 (or Argo CD).

In addition, you need OpenShift CLI version 4.3.0 or greater for Kustomize.

Finally, you can take advantage of the Web Terminal Operator if you prefer to use only a web browser for the entire deployment.

[ Learn how to build a flexible foundation for your organization. Download An architect's guide to multicloud infrastructure. ]


The following prerequisites must be in place to follow along with the steps in this article:

  • Install Argo CD or the Red Hat OpenShift GitOps operator on your Red Hat ACM for Kubernetes platform. I've included a policy to install the OpenShift Pipelines operator in my repo for this article. 
  • Import one or more managed clusters.
  • You'll need the ACM subscription administrator role to leverage all the automation components from the policies. (You need it even if you hold the cluster-admin role.) 

Implementing GitOps

The following image shows how ACM implements GitOps with the application subscription model.

Application subscription model with Git repository
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

The GitOps example in this tutorial is highlighted in the diagram above.

[ Download The path to GitOps to put the right tools, workflows, and structures in place to enable a complete GitOps workflow.  ] 

Subscriptions ( allow clusters to subscribe to the following source repositories (channels):

  • Git repository
  • Helm release registry
  • Object storage repository
Subscriptions model
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

Subscriptions use channels to identify new or updated resource templates. The subscription operator can download templates directly from the storage location and deploy them to targeted managed clusters without checking the hub cluster first. With a subscription, the subscription operator can monitor the channel for new or updated resources instead of using the cluster with the ACM Hub.

Deploy an application using the ACM web console

The administrator doing the deployment must have an access level that can create applications. You can only perform actions that are assigned to your role. The cluster-admin role can deploy the application, but if you need more granular access, review ACM's role-based access control documentation.

1. From the ACM web console, select ApplicationsCreate applicationSubscription.

Applications user interface
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

Select YAML: On to view the YAML in the console as you create your application (or you can import the YAML directly instead of using the interactive web form).

Create an app
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

2. The application in this example is called pacman. Enter the following values in the specified fields:

  • Name: pacman-app
  • Namespace: pacman

Select a namespace from the list. If you have the correct access role, you can also create a namespace using a valid Kubernetes name.

3. Choose Git from the list of repositories.

Create an application
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)
Reconcile the repository
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

If you want to learn more about the different options and how they work, refer to the Managing apps with Git repositories documentation.

[ For more on OpenShift, Argo CD, and Tekton, download the complimentary eBook Getting GitOps. ]

4. Click Save, and ACM redirects you to the app dashboard, where you can see all the objects that compose the app and if they are correctly deployed.

Deployed pacman-app
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)
pacman-app topology
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

Deploy Argo CD ApplicationSets in ACM

An ApplicationSet is an Argo CD subproject that adds multicluster support for Argo CD applications. You can create an ApplicationSet from the product console editor, as shown below.

[ A complimentary guide from Red Hat: The automation architect's handbook. ] 

Configure managed clusters for Argo CD

There are two approaches to configuring managed clusters for Argo CD:

  1. Follow the step-by-step procedure in the official Configuring managed clusters for OpenShift GitOps operator and Argo CD documentation.
  2. Install the policies in my repo for this article.

NOTE: Remember to grant your user the subscription administrator role before installing the policies.

If you prefer to continue with the automated approach, once the local-cluster complies with the policies, add the clusters you want to the all-openshift-clusters ClusterSet created by the policy-openshift-gitops-acm-integration policy.

You can choose any name for the ClusterSet, as ACM supports integration with non-OpenShift clusters. I used all-openshift-clusters in this case because the ClusterSet consists of OpenShift clusters alone. This is for simplicity, since the labelSelector of the Placement object, in charge of integrating the clusters in ArgoCD, is based on the label with the key/value vendor=OpenShift, which is the default label with any OpenShift cluster.

Application governance
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

1. Select ClustersCluster setsManage resource assignments.

Resource assignments
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

2. In the next screen, select the clusters, click Review, and then Save.

Confirm changes
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

You can confirm the assignment in two ways. The first method is by using the command interface:

$ argocd cluster list

[ Learn more about cloud-native development in the eBook Kubernetes Patterns: Reusable elements for designing cloud-native applications. ]

The other approach is by accessing the Argo CD web console at SettingsClusters:

managed clusters
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

The two managed clusters (highlighted in the screenshot) have been added to the list of clusters managed by Argo CD.

Deploy Argo CD ApplicationSets from ACM web console

Once you have enabled integration between OpenShift GitOps and ACM, you can deploy ApplicationSets with the ACM web console.

Start by deploying the ApplicationSet:

1. From the ACM web console, choose ApplicationsCreate applicationArgo CD ApplicationSet.

2. Navigate through the different steps on the web console's interactive form and fill in the relevant data. Here is an example:

Create an application set
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

Note that the destination namespace could be any value. It's not necessary for the ApplicationSet itself (but mandatory for the ACM form). It is better to leave the value bgdk to avoid misinterpretation because bgdk is the namespace that will generate the app deployment itself.

3. Clicking Create, and you are presented with the application. To check it, go to the Applications menu:

Create application
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

Drill down to see its evolution and whether all the elements are correctly deployed in both clusters by clicking on its name.

demo-appset overview
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)
demo-appset topology
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

In this case, select the Route object and check that it is deployed in both clusters and that you have direct access to the URL or can edit the object's YAML.

You can do the same with the Argo CD web console:

1. Check the general status.

general status
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

2. Drill down by clicking on one of the instances (one per cluster).

app-set instance
(Jose Antonio Gonzalez Prada, CC BY-SA 4.0)

Deploy with a YAML file

You can do the same as in the previous section using either the web console or the command line.

Create a YAML file with these contents:

kind: ApplicationSet
  name: acm-appsets
  namespace: openshift-gitops
    - clusterDecisionResource:
        configMapRef: acm-placement
        requeueAfterSeconds: 180
      name: 'acm-appset-'
        namespace: bgdk
        server: ''
      project: default
        path: apps/bgd/overlays/bgdk
        repoURL: ''
        targetRevision: single-app
          prune: true
          selfHeal: true

Here is the command-line example:

$ oc apply -f rhacm/deploy/acm-appset.yaml

Other methods include using the OpenShift web console (+AddImport YAML), Argo CD web console (+NEW APPEDIT AS YAML), or ACM web console (with the Create application menu).

Wrap up

At the beginning of this article, I outlined the challenge of deploying multicluster applications (whether in private, public cloud, or hybrid environments) while standardizing and centralizing critical functions across all environments. This includes:

  • Application lifecycle management
  • Governance
  • Observability
  • Multicluster lifecycle management

I have given more attention to the application lifecycle and governance part (thanks to ACM policies) in this article. For insight on multicluster management and observability, please consult the official documentation on these topics:

What to read next

Topics:   Kubernetes   OpenShift   Cloud  
Author’s photo

Jose Antonio Gonzalez Prada

Jose Antonio González Prada works for Red Hat as a Specialist Solution Architect, previously as a Senior Consultant, specializing in OpenShift and container-related technologies. More about me

Related Content