With Kubernetes Operators comes great responsibility

Operators are a powerful way to extend the functionality of Red Hat OpenShift Container Platform and Kubernetes. OpenShift provides features for deploying Operators in a safer way, such as OperatorHub, and the Operator Lifecycle Manager (OLM). In this post we explore safe ways to deploy Operators to Openshift 4.x using OperatorHub, OLM and scoping rules for Operators.

What are Operators?

Operators are a way to extend the functionality of OpenShift Container Platform. They can do day-to-day operations, such as installing software updates, as they continue to monitor the cluster after they are deployed. In OpenShift 4 we provide an OperatorHub and OLM to deploy Operators to a cluster. You can read more about Operators in the OpenShift 4 documentation.

Operators watch for the creation of custom or standard resources and react by modifying the cluster to some desired state, such as deploying pods. When installing an Operator via OperatorHub in the OpenShift web console, you have the option to install in a single namespace, or all namespaces. Which you choose usually depends on the Operator, but it can also depend on the permissions that Operator requires. 

Some Operators do not require cluster roles to manage the operand, especially if they are deployed in the same namespace as the operand. Therefore, deploying one of these Operators carries less risk of allowing an unintended escalation of privilege vulnerability. If you want an Operator to be able to deploy pods and other resources in any namespace, they will at least need a cluster role that gives permissions to create those resources, and usually other cluster permissions as well.

To create Custom Resource Definitions (CRD) within an OpenShift cluster, you need cluster-admin permissions. It is best if an OpenShift administrator first creates the CRD, checks that no other Operator is in control of the CRD, and creates any associated Service Accounts, Roles, and Role Bindings in the target namespace before allowing a regular user to deploy an Operator. OperatorHub in OpenShift takes care of this process for you. In fact, it goes a step further and deploys the Operator as well. This means that users other than the cluster-admin can safely install Operators if they are granted the ability via Role-Based Access Control (RBAC).

Let’s look at an example of deploying a Couchbase Operator to a single namespace by following the official documentation for OpenShift 4.2, then consulting the Couchbase Operator documentation.

1. Before following the documentation, create a namespace with the target regular user (developer):

$ oc login -u developer
$ oc new-project couchbase
Now using project "couchbase" on server …

2. Follow the documentation for OperatorHub to deploy the Operator to the target namespace created in the previous step. You’ll need to login as a user with cluster-admin permissions, e.g. system:admin.

3. Follow the Couchbase documentation to install a Couchbase cluster.

   1. Download the OpenShift Operator Package.

   2. Create the Couchbase Admin secret:

$ oc create -f secret.yaml
secret/cb-example-auth created

3. Create the Custom Resource:

$ oc create -f couchbase-cluster.yaml
couchbasecluster.couchbase.com/cb-example created

Now if you check for new cluster roles, you’ll see that only OLM cluster roles have been created:

$ oc get clusterroles | grep couchbase
couchbase-l7fqw-admin
couchbase-l7fqw-edit
couchbase-l7fqw-view
couchbaseclusters.couchbase.com-v1-admin
couchbaseclusters.couchbase.com-v1-crdview
couchbaseclusters.couchbase.com-v1-edit
couchbaseclusters.couchbase.com-v1-view

Now let’s look at another example of deploying an Operator to all namespaces. For an Operator to manage resources in all namespaces, they need to have permissions to control resources at a cluster scope. That can include the ability to create or update security settings, or view secrets across the entire cluster. Let’s take the AMQ Streams Operator and deploy it in the cluster using the all namespaces target.

1. Follow the documentation for OperatorHub to deploy the Operator to the openshift-Operators namespace. You’ll need to be logged in as a user with cluster-admin permissions, e.g. system:admin.

2. Login as a regular user (developer) and create a new project to deploy an AMQ Streams custom resource.

3. Follow the documentation to deploy a custom resource in that project.

When deploying privileged Operators, make sure they are deployed in namespaces that regular (non-admin) users do not have access. If you do, the regular user could use the Service Account token of the Operator to escalate their privileges on the cluster. By default, OperatorHub uses the OLM to deploy cluster-wide Operators to the openshift-Operators namespace. Do not give access to that namespace to non-privileged users.

If we now check for new cluster roles, we’ll see an amqstreams role has been created. You can view its permissions that are quite pervasive with this truncated list.

$ oc get clusterrolebindings | grep amq
amqstreams.v1.3.0-cfvn7-strimzi-cluster-Operator-prqtw

$ oc describe clusterrole/amqstreams.v1.3.0-cfvn7
Name:         amqstreams.v1.3.0-cfvn7
Labels:       olm.owner=amqstreams.v1.3.0
              olm.owner.kind=ClusterServiceVersion
              olm.owner.namespace=openshift-Operators
Annotations:  <none>
PolicyRule:
  Resources                                      Verbs
  ---------                                      -----
  …
clusterrolebindings.rbac.authorization.k8s.io    [get create delete patch update watch]
clusterroles.rbac.authorization.k8s.io           [get create delete patch ]
...
secrets                                          [get list create delete patch update]
...

Keep a check on what Operators can do

Only administrators should install Operators to a cluster. Let’s say you had another role in which you wanted to grant privileges to deploy Operators in a cluster. In order to keep a check on what permissions those Operators can request you can specify a service account to be used by an Operator during installation. For example, the serverless-Operator, from the Red Hat catalog, grants all permissions to all namespaces in a cluster. A cluster administrator might want to restrict that to all permissions in a single namespace. In that case, they can create a role, and bind that to a new service account for the Operator. For example:

$ oc new-project serverless-Operator

$ cat <<EOF | oc create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: scoped
  namespace: serverless-Operator
EOF

$ cat << EOF | oc create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: scoped
  namespace: serverless-Operator
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: scoped-bindings
  namespace: serverless-Operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: scoped
subjects:
- kind: ServiceAccount
  name: scoped
  namespace: serverless-Operator
EOF

Use of an Operator Group allows an administrator to restrict any Operators created in that namespace to only be deployed in a certain namespace, and only using the service account they specify:

$ cat <<EOF | oc create -f -
apiVersion: Operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: scoped
  namespace: serverless-Operator
spec:
  serviceAccountName: scoped
  targetNamespaces:
  - serverless-Operator
EOF

This lets the administrator restrict subscriptions created in the serverless-Operator group to only have the permissions granted to the scoped Service Account. This includes changes to the actual permissions granted, as well as the target namespace. 

An administrator could optionally allow other users to create subscriptions in the serverless-Operator namespace. But, in that case, they should restrict read permission of secrets in the serverless-Operator namespace in order to prevent those users from using Service Accounts created by those Operators to gain permissions of those Service Accounts.

Conclusion

Operators allow you to extend the functionality of OpenShift and Kubernetes. Operators can operate across an entire cluster, but usually require elevated permissions to do so. Be sure to review the permissions required by Operators that you are deploying to all namespaces and never allow an unprivileged user to read secrets in a namespace that has a cluster-wide Operator deployed.