Deploying Operators with OLM bundles

When operator-sdk generates or scaffolds out an operator, the Operator lacks logic about your application. It is up to the end user to include specifics for how to install and manage their app. Like any other operator, the resulting code is built into a container image. In this example, the operator image is named:

quay.io/username/doo-operator:v0.0.1

The Operator project structure created by the operator-sdk init command, will also create a Makefile that is then used throughout this post to perform various build and deployment tasks of the Operator.

This Operator image is built using the Makefile as shown in this example:

make docker-build docker-push IMG=quay.io/username/doo-operator:v0.0.1

Complete details on operator-sdk golang Operator scaffolding is found in this guide to building a Golang based Operator using Operator SDK.

Building the Operator Bundle Image

Apart from the Operator image itself, an Operator bundle is an OLM prescribed format for holding metadata about an operator. The metadata contains everything that Kubernetes needs to know in order to use the operator – its custom resource definitions (CRDs), Role-based access control (RBAC) roles and bindings required, dependency tree and other info. The operator-sdk generated Makefile enables you to create a bundle for your operator as follows:

make bundle IMG=quay.io/username/doo-operator:v0.0.1

This command generates an on-disk set of bundle manifests. Here is an example bundle directory structure that is generated:

bundle
├── manifests
│   ├── cache.example.com_dooes.yaml
│   ├── doo.clusterserviceversion.yaml
│   └── doo-metrics-reader_rbac.authorization.k8s.io_v1beta1_clusterrole.yaml
├── metadata
│   └── annotations.yaml
└── tests
    └── scorecard
        └── config.yaml

The make bundle command also creates a Dockerfile (bundle.Dockerfile) which is used to build a "bundle image." The bundle image is an OCI image that essentially holds the generated on-disk bundle manifest and metadata files.

In this example, we are naming the Operator bundle image as follows:

quay.io/username/doo-operator-bundle:v0.0.1

To create and push the Operator bundle image, run these Makefile targets:

make bundle-build BUNDLE_IMG=quay.io/username/doo-operator-bundle:v0.0.1

make docker-push IMG=quay.io/username/doo-operator-bundle:v0.0.1

Building the Operator Index Image

Another OLM concept is the "index image," which is like a catalog of Operators. This container image acts to serve an application programming interface (API) which describes information about your sample operator and others in the catalog. The index image includes information from your bundle image by running an operator-registry command, opm, as follows:

opm index add --bundles quay.io/username/doo-operator-bundle:v0.0.1 --tag quay.io/username/doo-operator-index:v0.0.1

podman push quay.io/username/doo-operator-index:v0.0.1

The index image holds a sqlite database with bundle definitions, it also runs a grpc service when the image is executed. The grpc service lets consumers query the sqlite database for information about the operators this index contains.

You can download the opm command from the operator-registry repository found on the GitHub operator-framework/operator-registry repository.

Running the Bundle Index Image

At this point, I am assuming you have OLM installed on your Kubernetes cluster. OLM is installed by default on Openshift clusters. Refer to the operator-sdk’s olm install command to manually install OLM on a non-Openshift cluster.

Once you have an index image, you deploy it by creating an OLM CatalogSource resource. For our sample operator, we create the following CatalogSource as follows:

cat <<EOF | kubectl create -f -
kind: CatalogSource
metadata:
  name: doo-operator
  namespace: operators
spec:
  sourceType: grpc
  image: quay.io/username/doo-operator-index:v0.0.1
EOF

​This CatalogSource is created in an existing namespace, created by OLM, named operators; on OpenShift clusters, this namespace is called openshift-operators. When you create the CatalogSource, it causes the index image to be executed as a Pod. You can view it as follows:

kubectl -n operators get pod --selector=olm.catalogSource=doo-operator
NAME                                                              READY   STATUS      RESTARTS   AGE
doo-operator-79x8z                                                1/1     Running     0          136m

You can look at the Pod log to make sure the image is serving the grpc API:

kubectl -n operators logs pod/doo-operator-79x8z
time="2020-10-05T13:17:04Z" level=info msg="Keeping server open for infinite seconds" database=/database/index.db port=50051
time="2020-10-05T13:17:04Z" level=info msg="serving registry" database=/database/index.db port=50051

Examining the Index Image

Once the index image is running, you can query it using a tool like grpcurl. For example, you can run these commands to directly access the grpc API:

kubectl -n operators port-forward pod/doo-operator-79x8z 50051:50051

In another terminal, run:

grpcurl -plaintext localhost:50051 api.Registry/ListPackages
{
  "name": "doo"
}

An alternative to the grpcurl command is to query the index image’s embedded sqlite database file. Notice that the database file is stored within the index image itself and is immutable. You can extract the sqlite database file used by the image using the following command:

kubectl -n operators cp doo-operator-79x8z:/database/index.db .

You can then use the sqlitebrowser UI to view the contents of the index database:

sqlitebrowser index.db

Using grpcurl or sqlitebrowser to view the index image’s contents is a means to debug what the operator description and manifests include and verify what version of an operator will be deployed when a Subscription is created.

Deploying

You use an OLM Subscription resource to trigger a specific operator deployment by OLM. The following command creates a Subscription which triggers the deployment of our sample operator:

cat <<EOF | kubectl create -f -
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: doo-subscription
  namespace: operators 
spec:
  channel: alpha
  name: doo
  source: doo-operator
  sourceNamespace: operators
EOF

Notice that the Subscription is created in the already existing operators namespace. This will cause OLM to create the sample operator into the operators namespace. Note, on Openshift clusters, the operators namespace is named openshift-operators. Keep in mind that a Subscription needs to be in the same namespace as your CatalogSource.

Verify

You can verify that the sample Operator is running using the following commands:

kubectl -n operators get subscription
NAME               PACKAGE   SOURCE         CHANNEL
doo-subscription   doo       doo-operator   alpha

kubectl -n operators get csv
NAME         DISPLAY        VERSION   REPLACES   PHASE
doo.v0.0.1   doo-operator   0.0.1                Succeeded

kubectl -n operators get pod
NAME                                      READY   STATUS    RESTARTS   AGE
doo-controller-manager-6c4bdf7db6-jcvpn   2/2     Running   0          10m

Test

You can test the sample operator by creating a CustomResource that the sample operator is watching. Here is an example of a CustomResource for the sample operator:

You can create the CustomResource in the default namespace as follows:

cat <<EOF | kubectl -n default create -f -
{
           "apiVersion": "cache.example.com/v1",
           "kind": "Doo",
           "metadata": {
             "name": "doo-sample"
           },
           "spec": {
             "foo": "bar"
           }
         }
 EOF

You should see that your sample operator, running in operators namespace, has responded to this CustomResource being created in default namespace by looking at the sample operator log:

kubectl -n operators logs pod/doo-controller-manager-6c4bdf7db6-jcvpn -c manager
2020-10-05T13:29:52.175Z DEBUG controller Successfully Reconciled{"reconcilerGroup": "cache.example.com", "reconcilerKind": "Doo", "controller": "doo", "name": "doo-sample", "namespace": "default"}

OperatorGroup

The examples shown in this blog utilize namespaces created by OLM when you install OLM. You might instead wish to isolate your operator deployments into a namespace that you create instead. For this, before you create your CatalogSource and Subscription, you will need to create a unique namespace (e.g. jeff-operators) and OperatorGroup (e.g. jeff-operators):

kubectl create namespace jeff-operators

cat <<EOF | kubectl create -f - 
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: jeff-operators
  namespace: jeff-operators
status:
  lastUpdated: "2020-10-07T13:44:54Z"
  namespaces:
  - ""
EOF

Details on how OperatorGroups work are found here, note that a CatalogSource and Subscription need to be created in a namespace that contains an OperatorGroup.

Future SDK OLM Bundle Automation

In upcoming operator-sdk releases, there is planned a run bundle command that performs many of the steps above using an temporary bundle index image. This new command should be useful for developers needing to test their operators using the OLM bundle architecture. Here are some examples of that new command:

operator-sdk run bundle quay.io/username/doo-operator-bundle:v0.0.1

Conclusion

The OLM bundle architecture provides an advanced mechanism to describe, publish, and deploy operators on k8s clusters.

Acknowledgements

Thank you to Eric Stroczynski and Jesus Rodriguez for reviewing this blog post.