Red Hat OpenShift and zero trust: Securing workloads with cert-manager and OpenShift Service Mesh

Version 1.15.1 might feel like a run-of-the-mill new release of cert-manager Operator for Red Hat OpenShift but actually it features more than a few notable enhancements to improve the security posture of your OpenShift clusters. This exciting release improves the overall security posture of your OpenShift clusters, and expands upon your ability to protect your cluster networking communications with TLS certificates managed by cert-manager. The release of Red Hat OpenShift 4.18 emphasises zero trust architecture, and introduces as Technology Preview, the powerful istio-csr agent via cert-manager’s new release.

The istio-csr agent enables a seamless integration with the Istio service mesh, allowing cert-manager to secure Istio workload and control plane components. The istio-csr agent is responsible for receiving all certificate signing requests from members of the Istio mesh and signing them with cert-manager. This integration between cert-manager and the Istio service mesh is notable as it further solidifies the security fundamentals provided by default with Istio. The Istio service mesh provides a number of security features in order to implement and adhere to the three zero trust principles. Cert-Manager adds an additional layer of security to the Istio security architecture by allowing cluster administrators to bring their own enterprise grade Certificate Authority(CA) to manage certificate authentication and authorization for the service mesh. 

Additionally, the general availability of Red Hat OpenShift Service Mesh v3.0 was recently announced, and is based on a Red Hat distribution of community Istio. This is exciting news as it means that cluster administrators can take advantage of this integration between cert-manager and Istio all within the comprehensive Red Hat OpenShift ecosystem. For simplicity, we will continue to discuss the cert-manager to Istio integration from the perspective of Istio. However, rest assured that OpenShift Service Mesh can be used synonymously with Istio in almost all cases.

Why you should use zero trust architecture

Traditional network security relies on the concept of perimeters. Essentially, trust is distributed to any agent who exists within a specific segment of the network. Imagine a network as a series of concentric rings, each ring consisting of at least one firewall protecting access from those outside its borders. However, any agent who is able to get past the firewall is then implicitly trusted by the network. In this case, if a malicious agent were to gain access to a highly secure part of the network, there could be disastrous effects.

This is where zero trust comes in. Derived from the simple concept of “never trust, always verify”, zero trust architecture is no longer a new concept in the world of cybersecurity, and its principles have existed in their current form for nearly a decade. The zero trust model sits on 3 core pillars:

• Continuous verification: For each interaction, verify users and devices
• Restricted access: Provide the least privileged access necessary
• Risk mitigation: Always assume there's a breach

This article focuses on the first and last of the core pillars because that's where the integration between Istio and cert-manager is most valuable.

The concept of continuous verification subscribes to the belief that all agents must be authenticated consistently while taking into account any number of relevant identifiers, including identity, location, and device type. This is opposed to the traditional approach of assigning trust based on the network segment the workload is originating from. The combination of cert-manager with an Istio service mesh tackles the issue of verifying the identity of any machine (service-to-service) workload. Within a service mesh, CAs are extremely important for workload identity verification. Forcing any applications on a network to verify their identity using cryptographic certificates, can significantly increase the security of your cluster. 

To some, it may feel like common sense to assume that there is always a security breach on the cluster, but often these breaches come as a complete surprise. Even with mTLS encryption, long-lived, manually managed certificates introduce vulnerabilities if a malicious agent is able to obtain a copy of the certificates being used, enabling it to impersonate a service within the mesh. However, shorter lived certificates introduce more complexity and increased workload to manage these certificates. The trade-off between accepting increased risk or committing to increased overhead makes for an extremely difficult decision for cluster administrators.

Istio service mesh security

The Istio service mesh is designed with enhanced security capabilities. This is valuable for those looking for a solution that does not require application code changes or significant amounts of fine tuning in order to defend against all possible security threats. At a high level, security capabilities from Istio can be broken down into several components, and deeper level documentation can be found in Istio's security documentation:

• Certificate Authority: Manages keys and certificates
• Configuration API server: Distributes authentication and authorization policies, as well as secure naming information to mesh proxies
• Sidecar and perimeter proxy: Acts as a policy enforcement point (PEP) to help secure client-server communication

The Certificate Authority (CA) plays a critical role in the security of the mesh. It's the core means by which an agent establishes its identity. However, Istio by default uses its own internal CA to create a self-signed root certificate. This private root key is then used to sign and issue leaf certificates which establish the identity and role of each workload in the service mesh. There are a few drawbacks to having a self-signed certificate, and Istio itself recommends using an external, trusted root CA. Additionally, Istio provides (through the configuration API server) the ability to define authentication and authorization policies for any service on the mesh, providing you with the ability to define explicit permissions for these services. These policies, in combination with mTLS, address the second pillar of zero trust.

The power of cert-manager and istio-csr

The ability to integrate cert-manager with an Istio service mesh, such as OpenShift Service Mesh, amplifies a cluster administrator’s ability to secure any workload on an OpenShift cluster. Building on its existing capabilities such as managing certificates for the ingress controller and apiserver, cert-manager extends its reach to even more critical areas of OpenShift’s network communications. This is valuable for large enterprises concerned with centralizing TLS certificate management and reducing reliance on multiple independent Certificate Authorities that must be managed for a cluster. In short, cert-manager provides two key features: the ability to bring in an external PKI to a service mesh, and the reduction of manual processes such as rotation of expired certificates. Additionally, the integration of cert-manager and Istio provides the added benefit of automation distribution of CA trust bundles throughout the service mesh, eliminating the need for additional add-on features such as cert-manager trust manager. Below is a reference diagram of the high-level architecture for an Istio service mesh integrated with cert-manager:

Security-driven private keys with external Certificate Authorities

From the perspective of zero trust, cert-manager offers significant enhancements to the default security features provided by the service mesh. Firstly, the private root key created by default for Istio is stored on-cluster as a Kubernetes Secret. This is generally not recommended as the private key is the method by which new leaf certificates are minted. While the Secret containing the private key is scoped to the privileged and isolated  istio-system namespace, this still presents an opportunity for an attacker who might be able to gain access to this secret.

In the default configuration, the istiod service operates as the Registration Authority and Certificate Authority, to approve and sign new certificate signing requests respectively. Leveraging cert-manager and istio-csr allows administrators to replace the default istiod with a new istio-agent service. As a result, it is now possible for the root CA to live externally of the service mesh environment. The istio-agent service forwards all signing requests to the intermediate CAs existing as part of the external secret manager. This further establishes the chain of trust and insulates the root CA from attacks. And cert-manager provides the ability to integrate with enterprise grade secrets managers like Cyberark Conjur, HashiCorp Vault, or CAs developed by the hyperscalers.

The architecture diagram above illustrates how a cluster can be configured to limit the impact of a breach within the service mesh, with red boxes serving to highlight spheres of isolation. The root CA can live in a separate namespace or even in a completely separate node or cluster, thus mitigating the impact of a breach within either system. Should the system containing the root CA be breached, the attacker would still need to find a way to access the network in which the mesh lives in order to take advantage of possessing the root private key. Similarly, if the mesh network is breached, the attacker would have no means to mint new leaf certificates because the root CA exists completely outside of the mesh. 

Additionally, the issuer configured with cert-manager can be instructed to authenticate itself by any method supported by the underlying secrets manager, meaning that an additional layer of protection can be implemented for workloads on the mesh. The root CA must still approve signing requests and sign the certificate, but now, the cert-manager Issuer must also authenticate any communication to the root CA with the authentication method expected by the external secret manager.

The below graphic shows a simplified flow of a certificate through a standalone Istio service mesh or OpenShift Service Mesh with cert-manager. This helps visualize the additional layers of security provided by the integration of cert-manager with the service mesh. 

Short-lived certificates and reduced manual processes

Use of short lived certificates can significantly improve the security posture of a cluster. Even if a breach were to occur, if the certificates are revoked and rotated consistently with new certificates, then the potential threat is reduced in correlation to the longevity of the certificate. However, the task of manually rotating certificates in the cluster comes with additional overhead and manual processes that could introduce risk for errors and downtime. As of September 2024, the community distribution of cert-manager became a graduated member of the Cloud Native Computing Foundation, and the automated management of large numbers of certificates across a cluster is one of the core value propositions of the software.

Cert-manager provides the ability to tune multiple aspects of the certificates it issues. Using a combination of spec.duration and spec.renewBefore you can dictate the lifespan of an issued certificate as well as the cadence at which it is reissued. By default, certificates issued by cert-manager have a lifespan of 1 hour compared to the default Istio certificate lifespan of 24 hours. Of course, this is not a solution to all security concerns, but cert-manager abstracts away a number of the difficulties associated with managing large numbers of short-lived certificates in a cluster. Cert-manager monitors the certificates it issues, and automatically rotates them, which means that cluster administrators can spend time focusing on setting proper policies rather than manually reissuing hundreds if not thousands of certificates in the cluster. 

Distribution of trust bundles, a symbiotic relationship

Finally, while the Istio service mesh benefits significantly from an integration with cert-manager, the same can be said for cert-manager. Trust bundles contain a complete set of all root and intermediate CAs used to sign leaf certificates, and play an integral part in securing a network and ensuring trust. Workloads with access to the trust bundle are capable of verifying the identity of any message sender by validating whether the sender's certificate was signed by a trusted CA and was not revoked. By default, Istio assigns any workload in the mesh a certificate representing its identity as well as a trust bundle used to recognize any identities in the mesh. Both cert-manager and the istio-csr feature leverage Istio's default behavior with the CA bundle distributor, enabling cert-manager to create the bundle initially and update it as new certificates are issued.

The cert-manager community did develop an additional operator called trust-manager. This is  capable of performing a similar distribution and update of trust bundles across a cluster. However, by inheriting Istio's default trust bundle distribution behavior, there is no longer a need to install and configure another operator, and workloads benefit from the Istio security features as mentioned above. This integration fills a gap for cert-manager without requiring additional resource consumption or the need to monitor new CRDs on the cluster. Trust-manager is designed to bolster the security of cert-manager, something that Istio performs by default along with the added security features of authentication and authorization policies.

Integration opportunities and what the future holds

Of course, the istio-csr feature for cert-manager cannot exist in isolation. It is designed to integrate with any Istio-based service mesh, making the integration with OpenShift Service Mesh the obvious candidate for integration with this release. Additionally, work is currently underway to productize the community SPIFFE/SPIRE project. The SPIRE project is a graduated member of the Cloud Native Computing Foundation, and focuses on secure methodologies for establishing workload identity. Requiring an issuer to authenticate itself adds an additional layer of security. Leveraging  SPIRE to establish trust between the cert-manager issuer and the external secret manager presents an intriguing integration opportunity, and warrants greater investigation.

In past releases, cert-manager has primarily been a single cluster operator. A cluster administrator could seamlessly manage distribution of certificates across a cluster originating from a single PKI, but dealing with cert-manager in multiple clusters was more complex. The Istio service mesh, and OpenShift Service Mesh, offers a number of 

multicluster topologies, and while further testing is needed, there is a clear opportunity for cert-manager to manage certificates across the entire multi-cluster mesh. We are actively working on High Availability features and improved scalability of the cert-manager operator to meet the new demands of a multi-cluster environment.