How to use Red Hat OpenShift GitOps with the Argo CD Vault Plugin as a sidecar

Sensitive data must not be stored in Git. This presents a conundrum for GitOps, because you need secrets in Git, but you don't want to expose this sensitive data! To address this problem, you can store secrets in a tool such as HashiCorp Vault, and then retrieve and inject that data into Red Hat OpenShift. 

This article is a guide for deploying OpenShift GitOps with the Argo CD Vault Plugin to retrieve secrets from a Hashicorp Vault instance and inject them into Kubernetes resources. This guide builds on work like this previous Red Hat blog from 2021 and is updated to use the more modern custom plugin sidecar approach.

Prerequisites

For this article, I assume you're running OpenShift 4.10, or greater, with administrative privileges on that cluster. I'm running a Red Hat OpenShift on AWS 4.13 cluster provisioned through the Red Hat demo system.

Additionally, ensure you're logged in to the cluster in a terminal environment with oc and helm binaries available before proceeding with any of the following steps. Use the following commands to confirm these binaries.

First, check prerequisites, you should see output similar to below:

$ oc version && helm version
Client Version: 4.11.13
Kustomize Version: v4.5.4
Kubernetes Version: v1.27.4
version.BuildInfo{Version:"v3.10.1", GitCommit:"9f88ccb6aee40b9a0535fcc7efea6055e1ef72c9", GitTreeState:"clean", GoVersion:"go1.18.7"}

1. Deploy Vault

For this article, I'm using Vault in dev mode, which is not suitable for production. You can skip this step if you already have a Vault instance running. Deploy an instance of Hashicorp Vault:

$ helm upgrade --install vault vault \
    --repo https://helm.releases.hashicorp.com \
    --namespace vault \
    --create-namespace \
    --set "global.openshift=true" \
    --set "server.dev.enabled=true" \
    --set "injector.enabled=false"

2. Configure Vault

Once you have deployed a Vault instance, run a shell script against the Vault pod to quickly configure it for this demo using the excellent vault CLI.

The configuration script sets the Kubernetes auth method for Vault, enables a kv-v2 secret store, creates a test secret, and creates an access policy for that secret.

Clone the source:

$ git clone https://github.com/jmhbnz/openshift-gitops-vault-plugin

Copy your configuration script to the Vault pod:

$ oc --namespace vault cp 1-vault/configure-vault.sh vault-0:/tmp/configure-vault.sh

Then run the script remotely in the vault pod:

$ oc --namespace vault exec vault-0 -t -- sh -c 'cd /tmp && ./configure-vault.sh'

3. Configure argocd-vault-plugin-auth to Vault

Once Vault is deployed and configured, we must enable the soon-to-be-deployed Argo CD Vault Plugin to authenticate to Vault using a Kubernetes service account.

Create a namespace for ArgoCD:

$ oc new-project vplugindemo

Create a service account for Argo Vault plugin authentication:

$ oc create serviceaccount vplugin

Create a role in Vault to bind your service account to the policy:

$ oc --namespace vault exec vault-0 -- vault write auth/kubernetes/role/vplugin \
    bound_service_account_names=vplugin \
    bound_service_account_namespaces=vplugindemo \
    policies=vplugin \
    ttl=1h

Finally, create a secret for the Argo Vault plugin to use when configuring the Vault connection.

$ oc --namespace vplugindemo create \
-f 2-argocd/secret-vault-configuration.yaml

4. Configure argocd-vault-plugin processing

With authentication configured, you now need to define what Argo CD Vault Plugin sidecar is used for. This is a two-step process. First, you must specify a discover command, then a generate command. Refer to these documented examples, including scenarios for helm or kustomize-based applications. My example takes the most basic approach of discovering files that contain an annotation, then uses argocd-vault-plugin generate . to template the files.

$ oc --namespace vplugindemo create \
-f 2-argocd/configmap-plugin.yaml

5. Deploy OpenShift GitOps

With Vault deployed and configured, and the argocd-vault-plugin configured, deploy the OpenShift GitOps operator:

$ oc --namespace openshift-operators create -f 2-argocd/subscription-gitops.yaml

Once the operator has been installed successfully, create your Argo CD instance using the custom resource:

$ oc --namespace vplugindemo create \
-f 2-argocd/crd-argocd.yaml

6. Create an Argo CD application

Once Argo CD is running, create this baseline sample application. This creates a secret resource on the cluster that has the default placeholder values replaced with values that come from Vault.

$ oc --namespace vplugindemo create \
-f 2-argocd/application-example.yaml

Summary

Using Hashicorp Vault, with Red Hat OpenShift GitOps and the Argo CD Vault Plugin running as a sidecar, you have successfully used placeholder values inside of a secret in Git. This is a great method of protecting sensitive data.