Requesting and installing Let’s Encrypt Certificates for OpenShift 4 on AWS

Overview

Red Hat OpenShift uses certificates to encrypt the communication with the Web Console as well as applications exposed as Routes. Without any further customization the install process will create self-signed certificates. While these work they usually trigger severe security warnings about unknown certificates in Web Browsers when accessing either the Web Console or any other application that is exposed via HTTPS.

Therefore for any Red Hat OpenShift cluster, it is suggested to use proper certificates to encrypt the routes and API endpoints. For many years, this required paying for the generation of those certificates and managing them across an organization. Today, thanks to Let’s Encrypt, this process can be automated and performed for free.

Luckily in OpenShift 4, it is reasonably straightforward to apply certificates after the installation has completed.

This blog walks through using Let’s Encrypt to provision certificates for your cluster in AWS. You will need to know the API endpoint URL and the Wildcard Domain for your ingress controller(s).

Installing acme.sh

If you already have certificates for your domains, you may skip this step and go straight to Installing Certificates for the Router.

In order to request Let’s Encrypt certificates we will use the acme.sh client. This client makes it very easy to request and update certificates.

1. Clone the acme.sh GitHub repository.
2. cd acme.sh
3. Update the file $HOME/acme.sh/dnsapi/dns_aws.sh with your AWS access credentials. This is necessary because you are requesting a certificate for a wildcard domain and Let’s Encrypt needs a way to validate that you are the owner of the wildcard domain.
4. Open the file in your favorite text editor and then add your AWS credentials. You will also need to remove the comment (#) before these two lines. The top of the file should look like this:

cd $HOME
git clone https://github.com/neilpang/acme.sh
#!/usr/bin/env sh
#
AWS_ACCESS_KEY_ID="YOUR ACCESS KEY"
#
AWS_SECRET_ACCESS_KEY="YOUR SECRET ACCESS KEY"
#This is the Amazon Route53 api wrapper for acme.sh
[...]

Requesting Certificates

1. Make sure that you are connected to your Red Hat OpenShift Cluster. You can either do these steps from a bastion host that you installed Red Hat OpenShift from or you can log into the cluster as a user that has cluster administrator permissions. Right after the installation this includes the system:admin and kubeadmin users.
2. To make things a bit easier, set two environment variables. The first variable should point to your API Endpoint. Use the oc CLI to find the API Endpoint URL.
3. oc whoami --show-server
4. Now set the variable LE_API to the fully qualified domain name:
5. Set the second variable LE_WILDCARD to your Wildcard Domain for example:
6. Run the acme.sh script. This will request one certificate which will be valid for both the API Server hostname and the default wildcard domain.
7. It is usually a good idea to move the certificates from the acme.sh default path to a well known directory. So use the --install-cert option of the acme.sh script to copy the certificates to $HOME/certificates.

Sample Output

https://cluster-e954-api.e954.ocp4.opentlc.com:6443
export LE_API=$(oc whoami --show-server | cut -f 2 -d ':' | cut -f 3 -d '/' | sed 's/-api././')
export LE_WILDCARD=$(oc get ingresscontroller default -n openshift-ingress-operator -o jsonpath='{.status.domain}')
${HOME}/acme.sh/acme.sh --issue -d ${LE_API} -d *.${LE_WILDCARD} --dns dns_aws
export CERTDIR=$HOME/certificates
mkdir -p ${CERTDIR}
${HOME}/acme.sh/acme.sh --install-cert -d ${LE_API} -d *.${LE_WILDCARD} --cert-file ${CERTDIR}/cert.pem --key-file ${CERTDIR}/key.pem --fullchain-file ${CERTDIR}/fullchain.pem --ca-file ${CERTDIR}/ca.cer

Installing Certificates for the Default Ingress Controllers

The following instructions work for OpenShift 4.6 and higher.

The ingress controller expects the certificates in a Secret. This secret needs to be created in the project openshift-ingress.

1. Use the following command to create the secret - and if you have existing certificates, make sure to provide the path to your certificates instead.
2. Now update the Custom Resource for your ingress controller. The default custom resource is of type IngressController, is named default and is located in the openshift-ingress-operator project. Note that this project is different from where you created the secret earlier.
3. This is all you need to do. After you update the IngressController object the OpenShift ingress operator notices that the custom resource has changed and therefore re-deploys the router.
4. Use the following command to create the secret - and if you have existing certificates, make sure to provide the path to your certificates instead.
5. Now update the Custom Resource for your API server. The custom resource is of type APIServer and is named cluster.
6. This is all you need to do. After you update the APIServer object the OpenShift API operator notices that the custom resource has changed and therefore re-deploys the API Servers. Note that this will take about 5 minutes per APIServer - which means about 15 minutes for a regular high availability deployment of OpenShift.

oc create secret tls router-certs --cert=${CERTDIR}/fullchain.pem --key=${CERTDIR}/key.pem -n openshift-ingress
oc patch ingresscontroller default -n openshift-ingress-operator --type=merge --patch='{"spec": { "defaultCertificate": { "name": "router-certs" }}}'

You now have proper certificates on the router - and this includes custom applications exposed using the default route, the Web Console for your OpenShift Cluster and various other services like the Monitoring stack. .

Installing Certificates for the API Endpoint

The API Server also expects the certificates in a Secret. This secret needs to be created in the project openshift-config.

oc create secret tls api-certs --cert=${CERTDIR}/fullchain.pem --key=${CERTDIR}/key.pem -n openshift-config
oc patch apiserver cluster --type merge --patch="{\"spec\": {\"servingCerts\": {\"namedCertificates\": [ { \"names\": [  \"$LE_API\"  ], \"servingCertificate\": {\"name\": \"api-certs\" }}]}}}"

You now have proper certificates on the API Endpoint. Note that depending on the age of your operating system you may still see an insecure connection error when using the `oc login` command - newer operating systems already trust the Let’s Encrypt certificate authority.