Operators are a powerful way to extend the functionality of Red Hat OpenShift Container Platform and Kubernetes. OpenShift provides features for deploying Operators in a safer way, such as OperatorHub, and the Operator Lifecycle Manager (OLM). In this post we explore safe ways to deploy Operators to Openshift 4.x using OperatorHub, OLM and scoping rules for Operators.
What are Operators?
Operators are a way to extend the functionality of OpenShift Container Platform. They can do day-to-day operations, such as installing software updates, as they continue to monitor the cluster after they are deployed. In OpenShift 4 we provide an OperatorHub and OLM to deploy Operators to a cluster. You can read more about Operators in the OpenShift 4 documentation.
Operators watch for the creation of custom or standard resources and react by modifying the cluster to some desired state, such as deploying pods. When installing an Operator via OperatorHub in the OpenShift web console, you have the option to install in a single namespace, or all namespaces. Which you choose usually depends on the Operator, but it can also depend on the permissions that Operator requires.
Some Operators do not require cluster roles to manage the operand, especially if they are deployed in the same namespace as the operand. Therefore, deploying one of these Operators carries less risk of allowing an unintended escalation of privilege vulnerability. If you want an Operator to be able to deploy pods and other resources in any namespace, they will at least need a cluster role that gives permissions to create those resources, and usually other cluster permissions as well.
To create Custom Resource Definitions (CRD) within an OpenShift cluster, you need cluster-admin permissions. It is best if an OpenShift administrator first creates the CRD, checks that no other Operator is in control of the CRD, and creates any associated Service Accounts, Roles, and Role Bindings in the target namespace before allowing a regular user to deploy an Operator. OperatorHub in OpenShift takes care of this process for you. In fact, it goes a step further and deploys the Operator as well. This means that users other than the cluster-admin can safely install Operators if they are granted the ability via Role-Based Access Control (RBAC).
Before following the documentation, create a namespace with the target regular user (developer):
$ oc login -u developer $ oc new-project couchbase Now using project "couchbase" on server …
Follow the documentation for OperatorHub to deploy the Operator to the target namespace created in the previous step. You’ll need to login as a user with cluster-admin permissions, e.g. system:admin.
Follow the Couchbase documentation to install a Couchbase cluster.
Download the OpenShift Operator Package.
Create the Couchbase Admin secret:
$ oc create -f secret.yaml secret/cb-example-auth created
Create the Custom Resource:
$ oc create -f couchbase-cluster.yaml couchbasecluster.couchbase.com/cb-example created
Now if you check for new cluster roles, you’ll see that only OLM cluster roles have been created:
$ oc get clusterroles | grep couchbase couchbase-l7fqw-admin couchbase-l7fqw-edit couchbase-l7fqw-view couchbaseclusters.couchbase.com-v1-admin couchbaseclusters.couchbase.com-v1-crdview couchbaseclusters.couchbase.com-v1-edit couchbaseclusters.couchbase.com-v1-view
Now let’s look at another example of deploying an Operator to all namespaces. For an Operator to manage resources in all namespaces, they need to have permissions to control resources at a cluster scope. That can include the ability to create or update security settings, or view secrets across the entire cluster. Let’s take the AMQ Streams Operator and deploy it in the cluster using the
all namespaces target.
Follow the documentation for OperatorHub to deploy the Operator to the
openshift-Operatorsnamespace. You’ll need to be logged in as a user with cluster-admin permissions, e.g.
Login as a regular user (developer) and create a new project to deploy an AMQ Streams custom resource.
Follow the documentation to deploy a custom resource in that project.
When deploying privileged Operators, make sure they are deployed in namespaces that regular (non-admin) users do not have access. If you do, the regular user could use the Service Account token of the Operator to escalate their privileges on the cluster. By default, OperatorHub uses the OLM to deploy cluster-wide Operators to the openshift-Operators namespace. Do not give access to that namespace to non-privileged users.
If we now check for new cluster roles, we’ll see an amqstreams role has been created. You can view its permissions that are quite pervasive with this truncated list.
$ oc get clusterrolebindings | grep amq amqstreams.v1.3.0-cfvn7-strimzi-cluster-Operator-prqtw $ oc describe clusterrole/amqstreams.v1.3.0-cfvn7 Name: amqstreams.v1.3.0-cfvn7 Labels: olm.owner=amqstreams.v1.3.0 olm.owner.kind=ClusterServiceVersion olm.owner.namespace=openshift-Operators Annotations: <none> PolicyRule: Resources Verbs --------- ----- … clusterrolebindings.rbac.authorization.k8s.io [get create delete patch update watch] clusterroles.rbac.authorization.k8s.io [get create delete patch ] ... secrets [get list create delete patch update] ...
Keep a check on what Operators can do
Only administrators should install Operators to a cluster. Let’s say you had another role in which you wanted to grant privileges to deploy Operators in a cluster. In order to keep a check on what permissions those Operators can request you can specify a service account to be used by an Operator during installation. For example, the serverless-Operator, from the Red Hat catalog, grants all permissions to all namespaces in a cluster. A cluster administrator might want to restrict that to all permissions in a single namespace. In that case, they can create a role, and bind that to a new service account for the Operator. For example:
$ oc new-project serverless-Operator $ cat <<EOF | oc create -f - apiVersion: v1 kind: ServiceAccount metadata: name: scoped namespace: serverless-Operator EOF $ cat << EOF | oc create -f - apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: scoped namespace: serverless-Operator rules: - apiGroups: ["*"] resources: ["*"] verbs: ["*"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: scoped-bindings namespace: serverless-Operator roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: scoped subjects: - kind: ServiceAccount name: scoped namespace: serverless-Operator EOF
Use of an Operator Group allows an administrator to restrict any Operators created in that namespace to only be deployed in a certain namespace, and only using the service account they specify:
$ cat <<EOF | oc create -f - apiVersion: Operators.coreos.com/v1 kind: OperatorGroup metadata: name: scoped namespace: serverless-Operator spec: serviceAccountName: scoped targetNamespaces: - serverless-Operator EOF
This lets the administrator restrict subscriptions created in the serverless-Operator group to only have the permissions granted to the
scoped Service Account. This includes changes to the actual permissions granted, as well as the target namespace.
An administrator could optionally allow other users to create subscriptions in the serverless-Operator namespace. But, in that case, they should restrict read permission of secrets in the serverless-Operator namespace in order to prevent those users from using Service Accounts created by those Operators to gain permissions of those Service Accounts.
Operators allow you to extend the functionality of OpenShift and Kubernetes. Operators can operate across an entire cluster, but usually require elevated permissions to do so. Be sure to review the permissions required by Operators that you are deploying to all namespaces and never allow an unprivileged user to read secrets in a namespace that has a cluster-wide Operator deployed.
About the author
Specializing in Kubernetes, container runtimes, and web applications, Jason Shepherd is a principal security engineer in Red Hat's Product Security team. With a passion for open source and dedication to client success, Shepherd is your go-to guy for security assessment and data for security audits.