Vault Integration Using Kubernetes Authentication Method

Introduction

Until recently, it wasn't straightforward for applications running on OpenShift to authenticate with Vault in order to retrieve secrets. It involved a complex orchestration workflow, involving multiple actors (including a Vault Controller and init container as outlined in this post) in order to retrieve secrets stored in Vault.

In this post, we demonstrate a simpler approach for applications to authenticate with Vault, in a way more native to Kubernetes. This is made possible using by using the Kubernetes authentication method that has been added (since Vault 0.8.3), to integrate Vault directly with Kubernetes.

Additionally, in this blog post, we demonstrate how to run Vault on OpenShift. We then configure Vault to use the Kubernetes Auth Method that can be used by client applications to authenticate with Vault. Finally, we demonstrate running a Spring Boot application client on OpenShift that connects to Vault using the Kubernetes Auth Method to retrieve secrets. It makes use of Spring Cloud Vault Config that supports this method out of the box.

Why Vault

Vault is a secret store software created by HashiCorp. With Vault you have a central place to manage external secret properties for your applications across all environments. Vault can manage static and dynamic secrets such as username/password and manage credentials for external services such as MySQL, PostgreSQL, Apache Cassandra, MongoDB, Consul, AWS and more. Vault can run in the cloud and also encrypts credentials at rest. Additionally, no OpenShift cluster admin can see the credentials, plus in Vault you can create sharded master keys so that no Vault admin can, by themselves, un-encrypt the credentials.

Kubernetes Auth Method

The Kubernetes authentication method can be used to authenticate with Vault using a Kubernetes Service Account Token. The token for a pod’s service account is automatically mounted within a pod at /var/run/secrets/kubernetes.io/serviceaccount/token and is sent to Vault for authentication. Vault is configured with a service account that has permissions to access the TokenReview API. This service account can then be used to make authenticated calls to Kubernetes to verify tokens of the service accounts of pods that want to connect to Vault to get secrets.
This is depicted in the diagram below.

1. Use the JWT token for a pod’s service account to authenticate with Vault. A file containing the token is automatically mounted at /var/run/secrets/kubernetes.io/serviceaccount/token.
2. Vault sends the service account token of a pod that wants to access it to the OpenShift master API for authentication. During this call, Vault uses the token of the service account with token reviewer permissions to authenticate with the master API. If the service account token of the pod is successfully authenticated, then a Vault token correctly scoped is returned to the pod.
3. The Vault token is subsequently used to retrieve the secrets from Vault. The Vault token is also short-lived: It's the application’s responsibility to renew it if new secrets need to be retrieved later.

Below are the steps to install Vault, enable the Kubernetes authentication method, and configure a Spring Boot application to authenticate with Vault using this method and retrieve secrets.

A. Install Vault

1. Requirements
You need the Vault CLI installed on your machine.<
2. Clone the repository below containing a Vault aware Spring Boot application that is using the Vault Kubernetes authentication method.

git clone https://github.com/raffaelespazzoli/credscontroller

3. Create a new project

oc new-project vault-controller

4. Install Vault.
Note: You need a cluster administrator role to execute the first step below. As you can see, we are adding the default serviceaccount to the anyuid SCC as the application needs to run as user root in the container. We then create a configmap that contains configuration data for Vault, and create the OpenShift service, deployment config, persistent volume claim that are defined in vault.yaml, and finally expose a reencrypt route on port 8200.

oc adm policy add-scc-to-user anyuid -z default

oc create configmap vault-config --from-file=vault-config=./openshift/vault-config.json

oc create -f ./openshift/vault.yaml

oc create route reencrypt vault --port=8200 --service=vault

5. Initialize Vault

export VAULT_ADDR=https://`oc get route | grep -m1 vault | awk '{print $2}'`

vault init -tls-skip-verify -key-shares=1 -key-threshold=1

Save the generated key and token.
6. Unseal Vault
You have to repeat this step every time you start Vault. Don't try to automate this step, this is manual by design. You can make the initial seal stronger by increasing the number of keys.
We will assume that the KEYS environment variable contains the key necessary to unseal Vault and that ROOT_TOKEN contains the root token.

For example:

export KEYS=tjgv5s7M4CtMeUz92dU9jV3EudPawgNz6euEnciZoFs=
export ROOT_TOKEN=1487cceb-f05d-63be-3e24-d08e429c760c
vault unseal -tls-skip-verify $KEYS
 

B. Setup Kubernetes Vault auth backend

1. Create a token reviewer service account called vault-auth in the vault-controller project

oc create sa vault-auth

2. Give the vault-auth service account permissions to create tokenreviews.authentication.k8s.io at the cluster scope

oc adm policy add-cluster-role-to-user system:auth-delegator system:serviceaccount:vault-controller:vault-auth

3. Get the token for the vault-auth service account

reviewer_service_account_jwt=$(oc serviceaccounts get-token vault-auth)

4. Get the internal OpenShift Certificate Authority from the pod running Vault.

pod=`oc get pods -n $(oc project -q) | grep vault | awk '{print $1}'`

oc exec $pod -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt >> ca.crt

5. Setup the Kubernetes auth backend
You need to have a Kubernetes mount in Vault per Kubernetes/OpenShift cluster you want it to connect to. Enable the Kubernetes authentication backend. If you don't use the -path variable, then it defaults to the path /auth/kubernetes. If you want to mount other Kubernetes backends, use the-path= variable... i.e. a -path=kubernetes/cluster1 will create a path /auth/kubernetes/cluster1

export VAULT_TOKEN=$ROOT_TOKEN
vault auth-enable -tls-skip-verify kubernetes
 

6. Configure the Kubernetes authentication method to use the vault-auth service account token to authenticate with the OpenShift master API, in order to verify other service account tokens of pods that want to access Vault.

vault write -tls-skip-verify auth/kubernetes/config token_reviewer_jwt=$reviewer_service_account_jwt kubernetes_host=<server-url> kubernetes_ca_cert=@ca.crt 
rm ca.crt
 

7. A sample policy file called spring-native-example.hcl has already been provided to you that allows a token to get a secret from the generic secret backend for the client role.

path "secret/application" {

  capabilities = ["read", "list"]

}

path "secret/spring-native-example" {

    capabilities = ["read", "list"]

}

8. Create the policy

vault policy-write -tls-skip-verify spring-native-example ./examples/spring-native-example/spring-native-example.hcl

9. Authorization with this backend is role based. Before a token can be used to login, it must be configured in a role.

vault write -tls-skip-verify auth/kubernetes/role/spring-native-example bound_service_account_names=default bound_service_account_namespaces='*' policies=spring-native-example ttl=2h

This authorizes all default service accounts (and pods running with the default service accounts) in all namespaces and gives them the spring-native-example policy, which allows them to read secrets from Vault.

10. Confirm that the policy enables secret creation only under the path secret/spring-native-example

export VAULT_TOKEN=$ROOT_TOKEN
vault write -tls-skip-verify secret/spring-native-example password=pwd
 

11. Verify that the backend can now be used to authenticate Vault requests using the default service account.
a. Get the default service account token from the default namespace

default_account_token=$(oc serviceaccounts get-token default -n default)

b. Log in to the Kubernetes auth backend using the service account token. The output returns a Vault token that can be subsequently used to read secrets.

vault write -tls-skip-verify auth/kubernetes/login role=spring-native-example jwt=${default_account_token}
Key Value
 --- -----
 token 74603479-607d-4ab8-a406-d0456d9f3d65
 token_accessor 4893b0a1-f42a-bfd8-cd9c-c14b9bdb6095
 token_duration 1h0m0s
 token_renewable true
 token_policies [default spring-native-example]
 token_meta_role "spring-native-example"
 token_meta_service_account_name "default"
 token_meta_service_account_namespace "default"
 token_meta_service_account_secret_name "default-token-fndln"
 token_meta_service_account_uid "aaf6c23c-b04a-11e7-9aea-0245c85cf1cc"
 

C. Configure Spring Boot Client application to Use the Kubernetes Auth Method to Authenticate with Vault

In this example, we are using Spring Cloud Vault in order to bind properties based on secrets. You need to have the following two prerequisites in your project.
1. Add the dependency on Spring Cloud Vault

<dependencyManagement>

    <dependencies>

        <dependency>

            <groupId>org.springframework.cloud</groupId>

            <artifactId>spring-cloud-vault-dependencies</artifactId>

            <version>1.1.0.RELEASE</version>

            <scope>import</scope>

            <type>pom</type>

        </dependency>

    </dependencies>

</dependencyManagement>
<dependencies>
 <dependency>
 <groupId>org.springframework.cloud</groupId>
 <artifactId>spring-cloud-starter-vault-config</artifactId>
 </dependency>
 </dependencies>
 

2. Configure Vault information in the bootstrap.yml file of the project

spring.application.name: spring-native-example

spring.cloud.vault:

    host: ${vault.host:localhost}

    port: ${vault.port:8200}

    scheme: https

    authentication: KUBERNETES

    kubernetes:

        role: spring-native-example

        service-account-token-file: /var/run/secrets/kubernetes.io/serviceaccount/token

The Kubernetes authentication mechanism is role-based and the role is bound to a service account name and namespace.

• role sets the Role.
• service-account-token-file sets the location of the file containing the Kubernetes Service Account Token. Defaults to /var/run/secrets/kubernetes.io/serviceaccount/token.

Deploy the Spring Boot application on OpenShift

1. Create the project

oc new-project spring-native-example

oc new-build registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift~https://github.com/raffaelespazzoli/credscontroller --context-dir=examples/spring-native-example --name spring-native-example

2. Join the project with vault-controller to be able to call the Vault service
Note: This is only required when using the multitenant SDN plugin

oc adm pod-network join-projects --to vault-controller spring-native-example

3. Deploy the spring-native-example application

oc create -f ./examples/spring-native-example/spring-native-example.yaml

oc expose svc spring-native-example

4. Now you should be able to call a service that returns the secret

export SPRING_EXAMPLE_ADDR=http://`oc get route | grep -m1 spring | awk '{print $2}'`

curl $SPRING_EXAMPLE_ADDR/secret

Summary

In this post, we saw how to run Vault on OpenShift and configure it to use the Kubernetes authentication method. We also showed how to deploy a reference Spring Boot application that makes use of this authentication method to authenticate with Vault and bind application properties to secrets stored in Vault.