Applying Automated Multicluster Governance to Optimize Operations of Cloud Platforms

Overview

Operating cloud platforms to meet enterprise standards and regulatory compliance requirements ensures that various controls for security, resiliency, and software engineering are configured and operating in the desired configuration state. Such a state is defined by best practices that conform to enterprise internal standards or regulatory compliance standards such as PCI, HIPAA, FISMA, or BSI-Grundschutz. Continue reading this blog to learn how policy management paired with an extensible configuration management can help achieve the goal of automating multicluster governance, as outlined in the Kubernetes Policy Workgroup White Paper. Kubernetes policy management can be implemented using Red Hat Advanced Cluster Management for Kubernetes (RHACM), a component of the Red Hat OpenShift Plus bundle.

NOTE: This policy management applies across the entire stack and deployment architecture including edge devices.

How RHACM and specific policy engines, like Gatekeeper, are related, is best explained using the architecture in the Kubernetes Policy Workgroup white paper. RHACM acts as the Policy Administration Point (PAP) and provides a way to create automated multicluster, policy-based governance, and both Red Hat and third party policy engines that serve as one of the policy enforcement points (PEPs).

Blended lines

RHACM defines pillars for various aspects to be considered when operating a cloud platform such as Application, Policy, Configuration, Monitoring, and Cluster Management. These are all managing workload resources in Kubernetes, which means the tooling that creates, applies, destroys, and modifies these resources actually works in all pillars. As an example, a policy can deploy an application and include the configuration of a cluster. You can deploy an application using GitOps Operator/ArgoCD when you use RHACM and Kubernetes plugins. Therefore, we want to clarify the high-level concepts and values that are delivered in each of these spaces, as the Kubernetes resource delivery mechanism is mostly commoditized.

Configuration management with Kubernetes

First, let us briefly review some common and well established concepts for configuration management in Kubernetes.

Configuration using YAML or JSON Files

Kubernetes is, by its nature, declarative, especially when using Operators. Therefore, you can configure instances in a declarative way. See the Red Hat community GitOps Catalog for a list of common Operators used by OpenShift.

Kustomize

Kustomize is a tool that customizes Kubernetes resources for different environments or needs. Kustomize is a template-free solution that helps reuse configurations and provides an easy way to patch any resource.

The principles of Kustomize are descried in the following list:

* Purely declarative approach to configuration customization
* Manage an arbitrary number of distinctly customized Kubernetes configurations
* Every artifact that Kustomize uses is plain YAML and can be validated and processed as such

Learn more by visiting Red Hat Developer - Working with Kustomize.

GitOps

While using a declarative approach it is reasonable to configure all artifacts in Git. You can have read access to the artifacts using GitOps like RHACM Application lifecycle management or ArgoCD - GitOps Operator.

Multicluster configuration management

Customers typically use more than one Kubernetes cluster, so being able to easily and consistently manage configuration across multiple clusters is a key requirement. Next, let's discuss the requirements to configure and manage multiple clusters.

Cluster placement

One of the most important concepts is configuring and deploying cluster artifacts that are being managed, such as applications and policies. Starting with version 2.4, RHACM supports a new Placement API, that provides enhanced features and is a foundational service of RHACM. Learn more by reading the Using the Open Cluster Management Placement for Multicluster Scheduling blog.

From configuration management to policy management

Policy management with configuration management refers to the requirements from organizational and regulatory bodies for operating various controls, such as policies that include how the controls and meta data needs to be configured. View the following list of some meta data parameters:

• Control, control category, and standard the policy applies to
• Severity to indicate risk of a non-compliance status
• Type of policy
• Defined actions when a policy is non-compliant
• Automations that are associated with a policy

Note, the upstream policies framework is part of the Open Cluster Management, which is a CNCF sandbox project.

Architecture

In this section, we outline the policy-based governance architecture of Red Hat Advanced Cluster Management for Kubernetes to illustrate how it can be used to achieve the Policy management best practices outlined in this Kubernetes policy management white paper. This Red Hat product uses the upstream Open-Cluster-Management technology.

In this section, learn more about the policy-based governance architecture of RHACM through an illustration on policy management best practices, which are outlined in the Kubernetes Policy Workgroup white paper. See the following diagram of the RHACM governance framework architecture:

Continue reading for descriptions of Policy Enforcement Points (PEP) techniques, which can be used to audit or enforce best practices for security, resiliency, and software engineering:

• Built-in RHACM policy controllers for configuration management and certificate management.
• Advanced Cluster Security for managing vulnerabilities, network policies, and least privilege principles.
• Compliance operator for auditing and enforcing security profiles required by various standards such as CIS and FISMA.
• Sandboxed-containers for isolating containers based on criticality of workloads
• Gatekeeper for admission control to enforce REGO-based policies, as well as to audit the policies.
• Kyverno for admission control to enforce declarative policies, as well as runtime policies.

As an enterprise solution, it is necessary to integrate with other systems, such as Ansible, metrics, and alert management.

Policy management benefits

Based on details discussed previously, see the following summary of the benefits of using policy management along with configuration management:

• Overall security posture.
• Assessment of degree of compliance to various standards (internal enterprise and regulatory compliance).
• Automated governance.
• Continuous audit readiness.
• Ease of collection of evidence for audits.
• Continuous security readiness.
• Ability to integrate with incident management, security operations center, and enterprise GRC tools.

Potential cost savings using policy management

Continuous security readiness leads to savings in operational costs. For example, ETCD encryption is automatically enabled on a production cluster because that cluster is provisioned with a label and an ETCD encryption policy enforce mode where both, for instance, have the env=prod label. Imagine the operational costs if an administrator has to manually enable this when a new production cluster is provisioned, while also having to periodically ensure that it is enabled? And that is one example of a security related control. Now imagine all the controls that need to be in place to effectively operate your fleet of clusters while adhering to standards for security, resiliency, and software engineering, along with the operational cost of doing so. RHACM governance allows you to automate that process and save those operational costs. It also constantly updates you on the operational state of environments.

Audit readiness

While it may still need review and customization, it helps cover the technical aspect required to prove compliance with organization rules to auditors.

Options to start with policy management using RHACM

Moving from declarative configuration management to policy management using the policy generator

Please consult the Generating Governance Policies Using Kustomize and GitOps blog for more information. We would like to highlight that you can use this approach to flexibly configure categories, standards, and controls you see in the schema of the policy generator.

Enhancing policy management by using policysets

Policies can be grouped together based on standards or other content. PolicySets are officially supported since RHACM 2.5 and you can review and use some examples.

Creating policies using the console

This is the direct way of pushing policies to a selected fleet of clusters. The console helps you select the correct template, where to place the policy, and provides basic validation.

Use and customize policy collection repository

For deploying a set of policies using GitOps-style, please follow the steps outlined here. Note that policies can also be deployed using ArgoCD/Gitops-Operator. Use the example, if you like to use RHACM/ArgoCD-Integration.

Use of templatized policies

Templatized policies are a powerful feature of RHACM to generate policies based on dynamic values coming from objects either on managed clusters or the hub cluster. In this example you can send audit logs to Kafka in a way that each cluster name represents a topic. In another example, you can have a secret management solution where you only need to maintain your secrets on a central point on the hub cluster. You reference them from the managed cluster so that those references to secrets can be easily stored in Git.

Metrics for policy violations

See details on how metrics can be generated out of policy violation, which in turn can be used to fine-tune operations.

Automated governance

RHACM governance supports automated governance, which means that once a policy violation is detected, Ansible is being invoked and Ansible (which knows the clusters that are affected), using our RHACM-Ansible-Collection, takes care of the remediation, making your Policy compliant again. There will be a blog on this toppic in the near future.

Deploying policies and personas

You can define a structure in Git so that policies can be grouped into PolicySets that can be authored by specific personas. View the following example description of a Git structure with a single repository and multiple folders:

• Security-related policies are organized into a security-hardening PolicySet folder (for example, ETCD encryption, roles, and rolebindings). The target persona is IT SecOps.
• Resiliency-related policies organized into a resiliency PolicySet folder (for example, liveness probe set for pods). The target persona is SRE/Admin.
• Any K8s resources and configuration organized into a configuration PolicySet folder. The target persona is SRE/Admin.
• Policies for specific policy engines (for example, Gatekeeper/OPA and Kyverno) are organized into individual PolicySet folders. The target persona here is SRE/Admin/IT SecOps.
• Operational policies organized into a HubOperational PolicySet (for example, deploying OPP, Ansible Automation Platform, RHOCP GitOps, cert-manager, external secrets, etc.). The target persona here is SRE/Admin.
• Create a deploy folder that includes RHACM subscriptions and ArgoCD applications needed to deploy policy sets defined in other folders. The target persona here is the SRE/Admin.

Conclusion

In this blog we outlined how policy management can be paired with configuration management to achieve automated multicluster governance, which is required to operate cloud platforms in a cost-efficient manner to meet enterprise and regulatory compliance requirements.