How to Integrate Open Policy Agent with Red Hat Advanced Cluster Management for Kubernetes policy framework

Overview

Open Policy Agent (OPA) is a generic policy engine to help you to make decisions based on the policy you defined using a lightweight programming language called Rego. Red Hat Advanced Cluster Management for Kubernetes policy framework provides a desired state-based management approach to inform and enforce the policy compliance of a set of clusters managed by Advanced Cluster Management for Kuberentes. In this article, we are going to demonstrate how to combine the power of enforcing Kubernetes resource admission controls with OPA policies in a flexible way.

The following products are used for the example in this article:

• Red Hat Advanced Cluster Management for Kubernetes (Installed on OpenShift 4.5.3)
• Open Policy Agent 0.12.0
• OpenShift or Kubernetes command-line interface (kubectl)

Installing OPA on Red Hat Advanced Cluster Management for Kubernetes

OPA provides well written documentation on how to install OPA on Kubernetes and integrate with Kubernetes admission control. In this article, the following instructions from the OPA Tutorial: Ingress Validation page are provided, with some smaller modifications for Advanced Cluster Management for Kubernetes.

To make it easier, access the source code from the GitHub repository, ch-stark/open-cluster-management-opa.git.

Complete the following steps to install OPA on Kubernetes and integrate with Kubernetes admission control:

1. Clone the Github repository. Run the following command to clone the ch-stark/open-cluster-management-opa.git repository:

   git clone git@github.com:ch-stark/open-cluster-management-opa.git

2. Configure oc or kubectl to point to the cluster that is managed by Advanced Cluster Management for Kubernetes.

   We are configuring contexts for the spoke and hub-clusters as described here https://openshift.tips/oc/.

3. Create a namespace for OPA. Run the following command:

   oc --context=spoke1 create namespace opa

4. Switch to the install folder by running the following command:

   cd open-cluster-management-opa/installopa 

   Generate the secret, or use existing Certificates as described in the OPA Kubernetes tutorial. Run the following command:

   oc --context=spoke1 create secret tls opa-server --cert=server.crt --key=server.key -n opa

5. Install OPA and configure the admission controller webhook in the opa namespace. Run the following command:

   oc --context=spoke1 apply -f admission-controller.yaml -n opa

6. Install webhook-configuration.yaml by running the following command:

   oc --context=spoke1 apply -f webhook-configuration.yaml -n opa

   Your resource might resemble the following content:

   cat > webhook-configuration.yaml <<EOF
   kind: ValidatingWebhookConfiguration
   apiVersion: admissionregistration.k8s.io/v1beta1
   metadata:
     name: opa-validating-webhook
   webhooks:
     - name: validating-webhook.openpolicyagent.org
       namespaceSelector:
         matchExpressions:
         - key: openpolicyagent.org/webhook
           operator: NotIn
           values:
           - ignore
       rules:
         - operations: ["CREATE", "UPDATE"]
           apiGroups: ["*"]
           apiVersions: ["*"]
           resources: ["*"]
       clientConfig:
         caBundle: $(cat ca.crt | base64 | tr -d '\n')
         service:
           namespace: opa
           name: opa
   EOF

When you run the command the following tasks are complete:

• Role and role bindings are created.

• An admission controller webhook is registered.

• A ServiceAccount is created with the appropriate permissions.

Now verify that the OPA containers have started by running the following commands:

oc --context=spoke1 logs opa-5cc86966dc-vz58s  -n opa -c kube-mgmt
time="2020-07-20T04:51:44Z" level=warning msg="First line of log stream." 
time="2020-07-20T04:51:44Z" level=info msg="Syncing extensions/v1beta1/ingresses." 
time="2020-07-20T04:51:44Z" level=info msg="Syncing v1/namespaces." 
time="2020-07-20T04:51:44Z" level=info msg="Listed extensions/v1beta1/ingresses and got 8 resources with resourceVersion 3279386. Took 29.367885ms." 
time="2020-07-20T04:51:44Z" level=info msg="Listed v1/namespaces and got 72 resources with resourceVersion 3279386. Took 38.48603ms." 
time="2020-07-20T04:51:44Z" level=info msg="Loaded extensions/v1beta1/ingresses resources into OPA. Took 11.393731ms. Starting watch at resourceVersion 3279386." 
time="2020-07-20T04:51:44Z" level=info msg="Loaded v1/namespaces resources into OPA. Took 24.48515ms. Starting watch at resourceVersion 3279386." 

oc --context=spoke1 logs opa-5cc86966dc-vz58s  -n opa -c opa | tail -10
"level": "info",
"msg": "Sent response.",
"req_id": 49389,
"req_method": "POST",
"req_path": "/",
"resp_bytes": 139,
"resp_duration": 1.055464,
"resp_status": 200,
"time": "2020-07-20T05:43:05Z"

If the containers fail, try to delete the pods from the opa-namespace again. Run the following command:

oc --context=spoke1 delete pods --all -n opa

Applying an Advanced Cluster Management for Kubernetes policy to enforce OPA policy

OPA deployments use a sidecar to load an OPA policy from the ConfigMap. Let's create a policy to distribute the ConfigMap on the selected spoke clusters:

1. Download the policy from the policy-collection repository. Run the following command:

   wget https://raw.githubusercontent.com/open-cluster-management/policy-collection/master/community/CM-Configuration-Management/policy-opa-sample.yaml

2. Modify the desired namespaces, and then create the policy-object. Run the following command to add the config-map into the opa-namespace:

   oc --context=hub create -f policy-opa-sample.yaml

   After it has been created, verify that the ConfigMap has been loaded by the OPA by running the following command:

   oc --context=spoke1 get cm nopod -n opa -o yaml | grep policy-status
   openpolicyagent.org/policy-status: '{"status":"ok"}'

   

Further explanation:

Red Hat Advanced Cluster Management for Kubernetes policy framework provides the capability to create any Kubernetes object on managed clusters by using an object-template. You can define one or more Kubernetes object in the object-templates section of your policy. When you set the complianceType parameter value to musthave and remediationAction to enforce, Advanced Cluster Management for Kubernetes policy framework checks to make sure the cluster it applies to has the defined objects created. In this example, a ConfigMap object is embedded which contains the OPA policy. This policy denies any pod, whose image URL does not start with hooli.com, to be created in the opa namespace.

Testing the results

Now it is time to see if the policy actually works.

Let’s try to create a pod that violates the policy. Let's create a PodPolicy for this test:

In this example, we are creating a pod in the default namespace, and in the opa namespace Your policy might resemble the following details:

apiVersion: policy.open-cluster-management.io/v1
kind: Policy
metadata:
  name: opa
  namespace: policies
  annotations:
    policy.open-cluster-management.io/categories: PR.PT Protective Technology
    policy.open-cluster-management.io/controls: PR.PT-3 Least Functionality
    policy.open-cluster-management.io/standards: NIST-CSF
spec:
  disabled: false
  policy-templates:
    - objectDefinition:
        apiVersion: policy.open-cluster-management.io/v1
        kind: ConfigurationPolicy
        metadata:
          name: opapolicy-sample-nginx-pod
        spec:
          namespaceSelector:
            exclude:
              - kube-*
            include:
              - default
          object-templates:
            - complianceType: musthave
              objectDefinition:
                apiVersion: v1
                kind: Pod
                metadata:
                  name: sample-nginx-pod
                  namespace: opa
                spec:
                  containers:
                    - name: nginx
                      image: 'nginx:1.7.9'
                      ports:
                        - containerPort: 80
            - complianceType: musthave
              objectDefinition:
                apiVersion: v1
                kind: Pod
                metadata:
                  name: sample-nginx-pod
                  namespace: default
                spec:
                  containers:
                    - name: nginx
                      image: 'nginx:1.7.9'
                      ports:
                        - containerPort: 80
          remediationAction: enforce
          severity: low
  remediationAction: enforce

From the console, the violation might appear similarly in the following screenshot:

Notice that the pod is only created in the default namespace. Run the following command to verify the namespace that the pod is in:

oc --context=kubespoke1 get pods -A |grep nginx
default sample-nginx-pod  1/1     Running     0          84s

Now, you have successfully integrated OPA with the Advanced Cluster Management for Kubernetes policy framework to do admission control on Kubernetes with your own OPA policy.

Conclusion

It this article, we walked through the steps required to integrate OPA with Red Hat Advanced Cluster Management for Kubernetes policy framework. We enabled the admission control using OPA engine on clusters that Advanced Cluster Management for Kubernetes manages by defining an Advanced Cluster Management for Kubernetes policy to enforce the creation of OPA policy. Finally, we tested the OPA policy by creating a pod that violates the policy rule.

References

• Kubernetes Admission Control
• Security Policy Overview