OpenShift GitOps - Enable your Developers to Deploy Applications Seamlessly​ Through ArgoCD

Satisfying Developer Requests

We have recently been in a situation where our customer wanted to just give access to the developers to manage the right projects in OpenShift. Developers at the same time, needed to create applications on OpenShift without needing to consult administrators for access to ArgoCD/OpenShift GitOps. This has increased in importance for non-production environments. If a developer team has gone through the effort of requesting budget, time, and resources to develop a new application on OpenShift, should they really wait for an administrator to change settings for them to get started? 

In this post we will examine the ability for administrators to create an environment in OpenShift GitOps that promotes the ability for developers to manage their own applications and content. This will speed up the development process, allowing developers the flexibility to deploy as frequently as they need to resources they have access to.

Recipe for success

Deploy GitOps. A basic deployment of GitOps in an OpenShift environment is necessary to do this successfully. More on how to do this can be found here Installing OpenShift GitOps - GitOps | CI/CD. The following custom resource definitions get installed 2 of which we are using below:

-bash $ oc get crd -o custom-columns=NAME:.metadata.name | grep argo
applications.argoproj.io
applicationsets.argoproj.io
appprojects.argoproj.io
argocds.argoproj.io

Once that has been completed, the next step is to ensure that administrator access has been given to your GitOps administrator team. In our example we have 2 teams, one being a platform admin team that has full access for managing Argo and the other being a read only team. The below illustrates an example of this:

apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  finalizers:
  - argoproj.io/finalizer
  name: openshift-gitops
  namespace: openshift-gitops
  ownerReferences:
  - apiVersion: pipelines.openshift.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: GitopsService
    name: cluster
spec:
***********
  rbac:
    defaultPolicy: ""
    policy: |
      p, role:platform-team-admin, applications, *, */*, allow
      p, role:platform-team-admin, *, *, *, allow
      p, role:platform-readonly, applications, get, */*, allow
      p, role:platform-readonly, repositories, get, *, allow
      p, role:platform-readonly, logs, get, *, allow
      p, role:platform-readonly, projects, get, *, allow
      p, role:platform-readonly, clusters, get, *, allow
      p, role:platform-readonly, accounts, get, *, allow
      p, role:platform-readonly, certificates, get, *, allow
      p, role:platform-readonly, gpgkeys, get, *, allow
      g, PlatformAdmins, role:platform-team-admin
      g, PlatformReaders, role:platform-readonly
    scopes: '[accounts,groups]'
***********

After the deployment of OpenShift GitOps, a list of custom resources has been imported in the OpenShift Cluster. These are the resources that could be created should a new team require access to OpenShift GitOps in order to create applications on OpenShift. The main resource that needs to be created is the AppProject. This is a realization of the ability of GitOps to create multi-tenanted environments. 

This would be a starting point for a ‘real life’ example on creating a project in GitOps to interact with a variety of namespaces in an OpenShift cluster:

kind: AppProject
metadata:
  name: teamproj
  namespace: openshift-gitops
spec:
  clusterResourceWhitelist:
  - group: '*'
    kind: '*'
  description: teamproj project
  destinations:
  - namespace: teamproj-dev
    server: https://kubernetes.default.svc
  - namespace: teamproj-uat
    server: https://kubernetes.default.svc
  - namespace: teamproj-pp
    server: https://kubernetes.default.svc
  namespaceResourceBlacklist:
  - group: '*'
    kind: NetworkPolicy
  - group: '*'
    kind: ResourceQuota
  - group: '*'
    kind: LimitRange
  roles:
  - description: Admin group for teamproj
    groups:
    - teamadmins
    name: team-admins
    policies:
    - p, proj:teamproj:team-admins, applications, *, teamproj/*, allow
    - p, proj:teamproj:team-admins, repositories, *, teamproj/*, allow
  - description: Readonly group for teamproj
    groups:
    - teamreaders
    name: teama-readers
    policies:
    - p, proj:teamproj:team-readers, applications, get, teamproj/*, allow
    - p, proj:teamproj:team-readers, repositories, get, teamproj

To allow for the namespaces mentioned above to be managed by OpenShift GitOps, we have labeled them with the following:

oc label namespace teamproj-dev argocd.argoproj.io/managed-by=openshift-gitops

What we see above are scoped resources for the project created in GitOps. Normally, an ArgoCD admin creates a project and decides in advance which clusters and Git repositories it defines. This is not possible for ever changing teams and clusters. It creates a problem in scenarios where a developer wants to add a repository or cluster after the initial creation of the project. The above project definition shows a reliable way of offering a self service definition for allowing developers to add their own repositories and applications in the GitOps area.

The above definition disallows changes to be made under the NetworkPolicy, ResourceQuota, and LimitRange resources of the OpenShift projects that the GitOps project is able to deploy to. This is a good way of ensuring that only cluster administrators are able to make those changes if the OpenShift RBAC matches the restrictions. This is useful in environments where there’s a very strict segregation of duty.

Moreover, it defines 2 separate teams of people that could be given access for both administrative and observability purposes in order to ensure the aforementioned segregation of duties.

The policy items reference Openshift-argo items as follows: openshiftGroup:argoGroup. Argo CD gets the info about the OpenShift user (like the user's group membership) from Dex, an identity service that uses OpenID Connect to drive authentication for other apps.

To test the above setup, we used https://github.com/argoproj/argocd-example-apps ‘guestbook’ application. This was successfully deployed on OpenShift and then we were able to observe said application with a user from the read-only group. 

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: openshift-gitops
spec:
  destination:
    namespace: teamproj-dev
    server: https://kubernetes.default.svc
  project: manual
  source:
    path: guestbook
    repoURL: https://github.com/argoproj/argocd-example-apps
    targetRevision: HEAD
  syncPolicy:
    syncOptions:
    - Validate=false

So currently, only applications, repositories and clusters can be project scoped, however, it seems apparent that there is demand for scoping other types of resources such as logs, as seen in https://github.com/argoproj/argo-cd/issues/7211. 

Food for thought

While individual manual project creation is where most people would start with, it would be recommended to proceed with an automation mechanism onboarding teams onto GitOps in such a way that enables a team to deploy to multiple environments in a cluster. This will allow the re-use of resources and would give team autonomy to enable the developers in creating promotion processes fitting to their development practices. 

We have automated the above using jinja templates and Ansible with a group of variables that are being parsed in per-team. This allows the automation to scale as when a new team is being introduced to the platform, a process runs to automate the creation of the items above. 

Related reading: https://cloud.redhat.com/blog/managing-applications-via-a-gitops-control-plane