Applications deployed and managed using the GitOps philosophy are often made of many files. There's Kubernetes manifests for Deployments, Services, Secrets, ConfigMaps, and many more which all go into a Git repository to be revision controlled. Argo CD, the engine behind the OpenShift GitOps Operator, then uses that Git repository as the source for the application. So, how do we define all that to Argo CD? Using the Application CRD.
An Argo CD Application is a representation of a collection of Kubernetes-native manifests (usually YAML), that makes up all the pieces of your application. An Application is a Custom Resource Definition (CRD), used to define an Application source type. The source type is a definition of which deployment tool is used (helm or git) and where those manifests are located. But there are some challenges with this.
The first challenge centers around bootstrapping. This is an issue of how do I deploy my GitOps applications in a “GitOps” friendly way? That is to say - how do I deploy my Argo CD Application CR manifest in a GitOps way, on day 0? Another challenge is how an Argo CD Application uses the source type. An Application can only have one type defined; this means that you can’t have an Application that has both a Helm Chart and YAML.
How do we deal with these challenges? In this blog, I will go over how to use ApplicationSets and go over different implementations of them.
App of Apps
Before I dive into ApplicationSets, I do want to take a moment to talk about the fact that the challenges were solved before. Many users opted to solve this issue by creating an Argo CD Application that deploys other Argo CD Applications.
Image taken from the Argo Project GitHub Pages Site
This method solved a lot of problems. It was a way for users to massively deploy applications in one shot - so instead of deploying hundreds of Argo CD Application objects, you just deploy one that deploys the rest for you. And, it was also a way of logically grouping real world applications that are made up of YAML manifests and Helm charts using Argo CD. This method also gave a convenient “watcher” Application, that made sure all Applications were deployed and healthy. You can read more about App of Apps from the Argo Project website.
This was a precursor to ApplicationSets.
ApplicationSets Overview
Argo CD ApplicationSets are an evolution of the “App of Apps” deployment pattern. It took the idea of “App of Apps” and expanded it to be more flexible and deal with a wide range of use cases. The ArgoCD ApplicationSets runs as its own controller and supplements the functionality of the Argo CD Application CRD.
ApplicationSets provide the following functionality:
- Use a single manifest to target multiple Kubernetes clusters.
- Use a single manifest to deploy multiple Applications from a single, or multiple, git repos.
- Improve support for monolithic repository patterns (also known as a “monorepo”). This is where you have many applications and/or environments defined in a single repository.
- Within multi-tenant clusters, it improves the ability of teams within a cluster to deploy applications using Argo CD (without the need for privilege escalation).
ApplicationSets interact with Argo CD by creating, updating, managing, and deleting Argo CD Applications. The ApplicationSets job is to make sure that the Argo CD Application remains consistent with the declared ApplicationSet resource. ApplicationSets can be thought of as sort of an “Application factory”. It takes an ApplicationSet and outputs one or more Argo CD Applications.
You can read more about ApplicationSets from the ApplicationSets documentation site.
Generators
What are generators?
The ApplicationSet controller is made up of “generators”. These “generators” instruct the ApplicationSet how to generate Applications by the provided repo or repos, and it also instructs where to deploy the Application. There are 3 “generators” that I will be exploring are:
- List Generator
- Cluster Generator
- Git Generator
Each “generator” tackles different scenarios and use cases. Every “generator” gives you the same end result: Deployed Argo CD Applications that are loosely coupled together for easy management. What you use would depend on a lot of factors like the number of clusters managed, git repo layout, and environmental differences.
List Generator
The “List Generator'', generates Argo CD Application manifests based on a fixed list. This is the most straightforward, as it just passes the key/value you specify in the elements section into the template section of the ApplicatonSet manifest. See the following example:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: bgd
namespace: openshift-gitops
spec:
generators:
- list:
elements:
- cluster: cluster1
url: https://api.cluster1.chx.osecloud.com:6443
- cluster: cluster2
url: https://api.cluster2.chx.osecloud.com:6443
- cluster: cluster3
url: https://api.cluster3.chx.osecloud.com:6443
template:
metadata:
name: '{{cluster}}-bgd'
spec:
project: default
syncPolicy:
automated:
prune: true
selfHeal: true
source:
repoURL: https://github.com/christianh814/gitops-examples
targetRevision: master
path: applicationsets/list-generator/overlays/{{cluster}}
destination:
server: '{{url}}'
namespace: bgd
Here each iteration of {{cluster}} and {{url}} will be replaced by the elements above. This will produce 3 Applications.
These clusters must already be defined within Argo CD, in order to generate applications for these values. The ApplicationSet controller does not create clusters.
You can see that my ApplicationSet deployed an Application to each cluster defined. You can use any combination of list elements in your config. It doesn’t have to be clusters or overlays. Since this is just a simple key/value pair generator, you can mix and match as you see fit.
Cluster Generator
Argo CD stores information about the clusters it manages in a Secret. You can list your clusters by looking at the list of your secrets in the openshift-gitops namespace.
$ oc get secrets -n openshift-gitops -l argocd.argoproj.io/secret-type=cluster
NAME TYPE DATA AGE
cluster-api.cluster1.chx.osecloud.com-74873278 Opaque 3 23m
cluster-api.cluster2.chx.osecloud.com-2320437559 Opaque 3 23m
cluster-api.cluster3.chx.osecloud.com-2066075908 Opaque 3 23m
When you use the argocd CLI to list these clusters, the controller reads the secret to glean the information it needs.
$ argocd cluster list
SERVER NAME VERSION STATUS MESSAGE
https://api.cluster1.chx.osecloud.com:6443 cluster1 1.20 Successful
https://api.cluster2.chx.osecloud.com:6443 cluster2 1.20 Successful
https://api.cluster3.chx.osecloud.com:6443 cluster3 1.20 Successful
The same is true for the ApplicationSet controller. It uses those same Secrets to generate parameters that will be used in the template section of the manifest. Furthermore, you can use label selectors to target specific configurations to specific clusters. You can then just label the corresponding secret. Here’s an example:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: bgd
namespace: openshift-gitops
spec:
generators:
- clusters:
selector:
matchLabels:
bgd: dev
template:
metadata:
name: '{{name}}-bgd'
spec:
project: default
syncPolicy:
automated:
prune: true
selfHeal: true
source:
repoURL: https://github.com/christianh814/gitops-examples
targetRevision: master
path: applicationsets/cluster-generator/overlays/dev/
destination:
server: '{{server}}'
namespace: bgd
Here, under .spec.generators.clusters you can see that I set the selector as bgd=dev. Any cluster matching this label will have the Application deployed to it. The {{name}} and {{server}} resources get populated by the corresponding name and server fields in the secret.
Initially, when I apply this, I won’t see anything.
If I take a look at the controller logs, you see that it shows that it generated 0 applications.
$ oc logs -l app.kubernetes.io/name=argocd-applicationset-controller | grep generated
time="2021-04-01T01:25:31Z" level=info msg="generated 0 applications" generator="&{0xc000745b80 0xc000118000 0xc000afa000 openshift-gitops 0xc0009bd810 0xc000bb01e0}"
This is because we used the label bgd=dev to indicate which cluster we want to deploy this Application. Let’s take a look at the secrets.
$ oc get secrets -l bgd=dev -n openshift-gitops
No resources found in openshift-gitops namespace.
Let’s label cluster1, and then verify it, to deploy this Application to that cluster.
$ oc label secret cluster-api.cluster1.chx.osecloud.com-74873278 bgd=dev -n openshift-gitops
secret/cluster-api.cluster1.chx.osecloud.com-74873278 labeled
$ oc get secrets -l bgd=dev -n openshift-gitops
NAME TYPE DATA AGE
cluster-api.cluster1.chx.osecloud.com-74873278 Opaque 3 131m
Taking a look at the UI, I should see the Application deployed.
Now to deploy this Application to another cluster, you can just label the secret of the cluster you want to deploy to.
$ oc label secret cluster-api.cluster3.chx.osecloud.com-2066075908 bgd=dev -n openshift-gitops
secret/cluster-api.cluster3.chx.osecloud.com-2066075908 labeled
$ oc get secrets -l bgd=dev -n openshift-gitops
NAME TYPE DATA AGE
cluster-api.cluster1.chx.osecloud.com-74873278 Opaque 3 135m
cluster-api.cluster3.chx.osecloud.com-2066075908 Opaque 3 135m
Now I have my Application deployed to both cluster1 and cluster3, and all I had to do was label the corresponding secret.
If you want to target all your clusters, you just need to set .spec.generators.clusters to an empty object {}. Example snippet below.
spec:
generators:
- clusters: {}
This will target all clusters that Argo CD has managed, including the cluster that Argo CD is running on, which is called the “in cluster”. Note that there are issues with this method and “in clusters”. Please see the following GitHub issue for more information.
Git Generator
The Git Generator takes how your Git repository is organized to determine how your application gets deployed.The Git Generator has two sub-generators: Directory and File.
Directory Generator
The Git Directory Generator generates the parameters used based on your directory structure in your git repository. The ApplicationSet controller will create Applications based on the manifests stored in a particular directory in your repository. Here is an example ApplicationSet manifest.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: pricelist
namespace: openshift-gitops
spec:
generators:
- git:
repoURL: https://github.com/christianh814/gitops-examples
revision: master
directories:
- path: applicationsets/git-dir-generator/apps/*
template:
metadata:
name: '{{path.basename}}'
spec:
project: default
syncPolicy:
automated:
prune: true
selfHeal: true
source:
repoURL: https://github.com/christianh814/gitops-examples
targetRevision: master
path: '{{path}}'
destination:
server: https://api.cluster1.chx.osecloud.com:6443
namespace: pricelist
This ApplicationSet deploys an Application that is made up of Helm charts and YAML working together. To understand how this works, it’s good to take a look at the tree view of my directory structure.
$ tree applicationsets/git-dir-generator/apps
applicationsets/git-dir-generator/apps
├── pricelist-config
│ ├── kustomization.yaml
│ ├── pricelist-config-ns.yaml
│ └── pricelist-config-rb.yaml
├── pricelist-db
│ ├── Chart.yaml
│ └── values.yaml
└── pricelist-frontend
├── kustomization.yaml
├── pricelist-deploy.yaml
├── pricelist-job.yaml
├── pricelist-route.yaml
└── pricelist-svc.yaml
3 directories, 10 files
The name of the application is generated based on the name of the directory, denoted as {{path.basename}} in the config, which is pricelist-config, pricelist-db, and pricelist-frontend.
The path to each application denoted as {{path}} will be based on what is defined under .spec.generators.git.directories.path in the config. Once I apply this configuration, it will show 3 Applications in the UI.
Now, when you add a directory with Helm charts or bare YAML manifests it will automatically be added when you push to the tracked git repo.
File Generator
The Git File Generator is the next subtype. This generator is also based on what is stored in your git repository but instead of directory structure, it will read a configuration file. This file can be called whatever you want, but it must be in JSON format. Take a look at my directory structure.
$ tree applicationsets/git-generator/
applicationsets/git-generator/
├── appset-bgd.yaml
├── base
│ ├── bgd-deployment.yaml
│ ├── bgd-namespace.yaml
│ ├── bgd-route.yaml
│ ├── bgd-svc.yaml
│ └── kustomization.yaml
├── cluster-config
│ ├── cluster1
│ │ └── config.json
│ ├── cluster2
│ │ └── config.json
│ └── cluster3
│ └── config.json
└── overlays
├── cluster1
│ ├── bgd-deployment.yaml
│ └── kustomization.yaml
├── cluster2
│ ├── bgd-deployment.yaml
│ └── kustomization.yaml
└── cluster3
├── bgd-deployment.yaml
└── kustomization.yaml
Take note that this structure includes a cluster-config directory. In this directory there is a config.json file that contains information about how to deploy the application by providing the information needed to pass to the template in the ApplicationSets manifest. You can configure the config.json file as you like, as long as it’s valid JSON. Here is my example for cluster 1.
{
"cluster": {
"name": "cluster1",
"server": "https://api.cluster1.chx.osecloud.com:6443",
"overlay": "cluster1"
}
}
Based on this configuration, I can build the ApplicationSet YAML.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: bgd
namespace: openshift-gitops
spec:
generators:
- git:
repoURL: https://github.com/christianh814/gitops-examples
revision: master
files:
- path: "applicationsets/git-generator/cluster-config/**/config.json"
template:
metadata:
name: '{{cluster.name}}-bgd'
spec:
project: default
syncPolicy:
automated:
prune: true
selfHeal: true
source:
repoURL: https://github.com/christianh814/gitops-examples
targetRevision: master
path: applicationsets/git-generator/overlays/{{cluster.overlay}}
destination:
server: '{{cluster.server}}'
namespace: bgd
This configuration takes the configuration files you’ve stored, which is denoted under .spec.generators.git.files.path section, and reads the configuration files to use as parameters for the template section.
Using the configuration file instructs the controller how to deploy the application, as you can see from the screenshot. This is the most flexible of all, since it’s based on what you put in the JSON configuration.
Currently, only JSON is the supported format. You can track the progress of supporting YAML based configurations following this GitHub Issue.
Conclusion
In this blog we went over some of the challenges of deploying applications with Argo CD. We briefly went over how the “App of Apps” pattern tried to solve this problem. Finally, we explored ApplicationSets and the various implementations of it that expanded on the lessons learned from the “App of Apps” pattern. For more information about ApplicationSets, you can visit the documentation site.
Curious about what OpenShift GitOps has to offer? Then take a look at our announcement blog! To learn more about GitOps in general, please make sure to tune in to GitOps Guide to the Galaxy on OpenShift.TV! We stream live every other Thursday at 3pm Eastern Time. You can catch up on past shows by visiting https://red.ht/gitops.
About the author
Christian Hernandez currently leads the Developer Experience team at Codefresh. He has experience in enterprise architecture, DevOps, tech support, advocacy, software engineering, and management. He's passionate about open source and cloud-native architecture. He is an OpenGitOps Maintainer and an Argo Project Marketing SIG member. His current focus has been on Kubernetes, DevOps, and GitOps practices.
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Customer support
- Developer resources
- Find a partner
- Red Hat Ecosystem Catalog
- Red Hat value calculator
- Documentation
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit