Ask an OpenShift Admin Office Hour - SSL Certificates and OpenShift

Over the last 10+ years, the use of SSL and TLS certificates to encrypt and verify the authenticity of traffic between a client and server has become ubiquitous. OpenShift is no exception. Certificates are used for communication between worker nodes and the control plane, between control plane nodes, and for many different components to communicate with each other. However, for most of us, the most obvious use of certificates with OpenShift is when we connect to the API and ingress endpoints. 

In this week’s stream, we are joined by Brian Beaudoin to look into all the different ways certificates are used in OpenShift, where they come from, how they’re lifecycled, how to replace the certificate authority, and other interesting aspects of the certificates used all across OpenShift.

As always, please see the list below for additional links to specific topics, questions, and supporting materials for the episode!

If you’re interested in more streaming content, please subscribe to the OpenShift.tv streaming calendar to see the upcoming episode topics and to receive any schedule changes. If you have questions or topic suggestions for the Ask an OpenShift Admin Office Hour, please contact us via Discord, Twitter, or come join us live, Wednesdays at 11am EDT / 1500 UTC, on YouTube and Twitch.

Episode 32 recorded stream:

Use this link to jump directly to where we start talking about today’s topic.

This week’s top of mind topics:

• VMware recently published a new reference architecture for deploying OpenShift to VMware Cloud Foundation using Dell/EMC VxRail hardware. We also talk about the overall status of OpenShift reference architectures and why you don’t see Red Hat creating and publishing them. Importantly, you can find a consolidated - but incomplete - list of partner reference architectures here.
• During episode 30 we talked about subscriptions and entitlements, including that the Red Hat developer entitlement also gives you access to no-cost OpenShift. However, we’ve since discovered that there was an error in the backend Red Hat system which prevents the clusters from being entitled. That issue is being fixed, so please check back if you’re having issues! If you want details about the OpenShift developer entitlement, check Appendix 1 to the user agreement, which includes the precise conditions and caveats which apply.
• VMware Site Recovery Manager and Fault Tolerance can be used with OpenShift, without invalidating support for the cluster, but be cognizant of the ramifications. For example, performance impacts to etcd (which we talked about during episode 21).

Questions answered during the stream:

• Is there a difference in OpenShift’s self-signed certificates and other certificates used in the cluster? No, it comes down to what the certs are used for and how often they’re rotated. Some certs expire as quickly as four hours, so using an internal signing authority makes it easier for intra-cluster traffic.\
• Are self-signed certificates used externally? By default, yes. The routes will use a self-signed certificate unless you configure them differently, either manually or using a third-party solution.
• What and where are certificates used in OpenShift? There’s a lot of them, used for communication between nodes, pods, clients, and more.
• We had a good discussion about pod-to-pod traffic being encrypted at the Service Mesh vs node layer (via OVN-Kubernetes IPSec feature).
• What is cert-manager and how / when do I use it with OpenShift? Cert-manager is a third-party integration which dynamically requests certificates from Let’s Encrypt for Routes. You can find more details here.
• Brian talks about the differences between the ingress controller route settings for pass-through and re-encrypting traffic, including some scenarios where you might want to use one or the other.
• What is the purpose of kube-apiserver-to-kubelet-signer in OpenShift 4? This is one of the signing authorities used for internal node-to-node traffic.
• When approving certificate signing requests (CSRs) for nodes joining the cluster, what’s actually being requested and approved? This is, more or less, a human verifying that the node joining is the node they expect it to approve.
• Clusters can be shutdown gracefully, but sometimes when they come back the certificates need to be reapproved. What’s the time limit and what’s happening behind the scenes? The certificates have expired, so the CSR reapproval is to revalidate that they are the nodes expected.
• Can the expiration date for certificates be extended? For example, in a lab I don’t want to have to worry about them expiring and having to manually intervene. This is probably technically possible, but there are many other reasons why you wouldn’t want to do it, even in a lab.
• If you’re planning on shutting down a cluster for a long period of time, it’s a good idea to make sure you have the kubeconfig for the system:admin user, which can be recreated if you lost it.
• How many CAs are used in OpenShift? More than you might think, this blog post has additional details.
• If using the re-encrypt strategy with the ingress controller, does the router’s certificate need to be provided to the config? No, the CA will be made accessible to the pod automatically.
• You can use a Secret for a Route’s certificate - sort of. This works by creating an Ingress which uses the Secret and the Route Operator will automatically convert it.
• What are the best utilities to use for debugging issues with certificates? Brian has a lot of great advice here, including using openssl_s_client. Give this segment a listen if you’re having issues with certificate SNI, hostnames, etc.! He points out, specifically, that some problems are a result of OpenShift’s strict implementation of the TLS spec, such as the order of certificates that the client must send.