Deploying your storage backend using OpenShift Container Storage 4

1. Overview

This Blog is for both system administrators and application developers interested in learning how to deploy and manage Red Hat OpenShift Container Storage 4 (OCS). This Blog outlines how you will be using OpenShift Container Platform (OCP) 4.2.14+ and the OCS operator to deploy Ceph and the Multi-Cloud Object Gateway as a persistent storage solution for OCP workloads. If you do not have a current OpenShift test cluster, you can deploy OpenShift 4 by going to the OpenShift 4 Deployment page and then follow the instructions for AWS Installer-Provisioned Infrastructure (IPI).

> Note: If deploying OpenShift 4 on VMware infrastructure or other User-Provisioned Infrastructure (UPI) skip the steps that include creating machines and machinesets given these resources are not currently available for UPI. To complete all of the instructions in this lab you will need to pre-provision six OCP worker nodes if OCP 4 is on UPI (i.e. not on AWS). To deploy OCS, OCP worker nodes need to have a minimum of 16 CPUs and 64 GB memory available.

1.1. In this Blog you will learn how to

• Configure and deploy containerized Ceph and NooBaa
• Validate deployment of containerized Ceph Nautilus and NooBaa
• Deploy the Rook toolbox to run Ceph and RADOS commands

Figure 1. OpenShift Container Storage components

2. Deploy your storage backend using the OCS operator

2.1. Scale OCP cluster and add 3 new nodes

In this section, you will first validate the OCP environment has 3 master and 2 worker nodes (or 3 worker nodes) before increasing the cluster size by additional 3 worker nodes for OCS resources. The NAME of your OCP nodes will be different than shown below.

oc get nodes

Example output:

NAME STATUS ROLES AGE VERSION

ip-10-0-135-157.ec2.internal Ready worker 38m v1.14.6+c07e432da

ip-10-0-138-253.ec2.internal Ready master 42m v1.14.6+c07e432da

ip-10-0-157-189.ec2.internal Ready master 42m v1.14.6+c07e432da

ip-10-0-159-240.ec2.internal Ready worker 38m v1.14.6+c07e432da

ip-10-0-164-70.ec2.internal Ready master 42m v1.14.6+c07e432da

Now you are going to add 3 more OCP compute nodes to cluster using machinesets.

oc get machinesets -n openshift-machine-api

This will show you the existing machinesets used to create the 3 worker nodes in the cluster already. There is a machineset for each AWS AZ (us-east-1a, us-east-1b, us-east-1c). Your machinesets NAME will be different than below.

NAME DESIRED CURRENT READY AVAILABLE AGE

cluster-ocs-79cf-lj8wq-worker-us-east-1a 1 1 1 1 43m

cluster-ocs-79cf-lj8wq-worker-us-east-1b 1 1 1 1 43m

cluster-ocs-79cf-lj8wq-worker-us-east-1c 0 0 43m

cluster-ocs-79cf-lj8wq-worker-us-east-1d 0 0 43m

cluster-ocs-79cf-lj8wq-worker-us-east-1e 0 0 43m

cluster-ocs-79cf-lj8wq-worker-us-east-1f 0 0 43m

> Warning: Make sure you do the next step for finding and using your CLUSTERID

CLUSTERID=$(oc get machineset -n openshift-machine-api -o jsonpath='{.items[0].metadata.labels.machine\.openshift\.io/cluster-api-cluster}')

echo $CLUSTERID

curl -s https://raw.githubusercontent.com/red-hat-storage/ocs-training/master/ocp4ocs4/cluster-workerocs.yaml | sed "s/CLUSTERID/$CLUSTERID/g" | oc apply -f -

Check that you have new machines created.

oc get machines -n openshift-machine-api

They may be in pending for sometime so repeat command above until they are in a running STATE. The NAME of your machines will be different than shown below.

NAME STATE TYPE REGION ZONE AGE

cluster-ocs-79cf-lj8wq-master-0 running m4.xlarge us-east-1 us-east-1a 54m

cluster-ocs-79cf-lj8wq-master-1 running m4.xlarge us-east-1 us-east-1b 54m

cluster-ocs-79cf-lj8wq-master-2 running m4.xlarge us-east-1 us-east-1c 54m

cluster-ocs-79cf-lj8wq-worker-us-east-1a-xscbs running m4.4xlarge us-east-1 us-east-1a 54m

cluster-ocs-79cf-lj8wq-worker-us-east-1b-qcmrl running m4.4xlarge us-east-1 us-east-1b 54m

cluster-ocs-79cf-lj8wq-workerocs-us-east-1a-xmd9q running m4.4xlarge us-east-1 us-east-1a 46s

cluster-ocs-79cf-lj8wq-workerocs-us-east-1b-jh6k4 running m4.4xlarge us-east-1 us-east-1b 46s

cluster-ocs-79cf-lj8wq-workerocs-us-east-1c-649kq running m4.4xlarge us-east-1 us-east-1c 45s

You can see that the workerocs machines are using are also using the AWS EC2 instance type m4.4xlarge. The m4.4xlarge instance type follows our recommended instance sizing for OCS, 16 cpu and 64 GB mem.
Now you want to see if our new machines are added to the OCP cluster.

watch oc get machinesets -n openshift-machine-api

This step could take more than 5 minutes. The result of this command needs to look like below before you proceed. All new workerocs machinesets should have an integer, in this case 1, filled out for all rows and under columns READY and AVAILABLE. The NAME of your machinesets will be different than shown below.

NAME DESIRED CURRENT READY AVAILABLE AGE

cluster-ocs-79cf-lj8wq-worker-us-east-1a 1 1 1 1 62m

cluster-ocs-79cf-lj8wq-worker-us-east-1b 1 1 1 1 62m

cluster-ocs-79cf-lj8wq-worker-us-east-1c 0 0 62m

cluster-ocs-79cf-lj8wq-worker-us-east-1d 0 0 62m

cluster-ocs-79cf-lj8wq-worker-us-east-1e 0 0 62m

cluster-ocs-79cf-lj8wq-worker-us-east-1f 0 0 62m

cluster-ocs-79cf-lj8wq-workerocs-us-east-1a 1 1 1 1 8m26s

cluster-ocs-79cf-lj8wq-workerocs-us-east-1b 1 1 1 1 8m26s

cluster-ocs-79cf-lj8wq-workerocs-us-east-1c 1 1 1 1 8m25s

You can exit by pressing Ctrl+C
Now check to see that you have 3 new OCP worker nodes. The NAME of your OCP nodes will be different than shown below.

oc get nodes -l node-role.kubernetes.io/worker

Example output:

NAME STATUS ROLES AGE VERSION

ip-10-0-131-236.ec2.internal Ready worker 4m32s v1.14.6+c07e432da

ip-10-0-135-157.ec2.internal Ready worker 60m v1.14.6+c07e432da

ip-10-0-145-58.ec2.internal Ready worker 4m28s v1.14.6+c07e432da

ip-10-0-159-240.ec2.internal Ready worker 60m v1.14.6+c07e432da

ip-10-0-164-216.ec2.internal Ready worker 4m35s v1.14.6+c07e432da

2.2. Installing the OCS operator

In this section you will be using three of the worker OCP 4 nodes to deploy OCS 4 using the OCS Operator in OperatorHub. The following will be installed:
Groups and sources for the OCS operators
An OCS subscription
All OCS resources (Operators, Ceph pods, Noobaa pods, StorageClasses)
Start with creating the openshift-storage namespace.

oc create namespace openshift-storage

You must add the monitoring label to this namespace. This is required to get prometheus metrics and alerts for the OCP storage dashboards. To label the openshift-storage namespace use the following command:

oc label namespace openshift-storage "openshift.io/cluster-monitoring=true"

> Note: The below manifest file will not be needed when OCS 4 is released.

To apply this manifest, execute the following:

oc apply -f https://gist.githubusercontent.com/netzzer/207e00a1cbc86006652a100d28be9987/raw/ea441f97a1bfdf476d756bf986b8038ceb086076/deploy-with-olm.yaml

Now switch over to your Openshift Web Console. You can get your URL by issuing command below to get the OCP 4 console route. Put this URL in a browser tab. You will use the same Admin username and password you used to login and use the oc client to login to the OCP 4 console.

oc get -n openshift-console route console

Once you are logged in, navigate to the OperatorHub menu.

Figure 2. OCP OperatorHub

Now type container storage in the Filter by keyword…​ box.

Figure 3. OCP OperatorHub filter on OpenShift Container Storage Operator

Select OpenShift Container Storage Operator and then select Install.

Figure 4. OCP OperatorHub Install OpenShift Container Storage
On the next screen make sure the settings are as shown in this figure. Make sure to change to A specific namespace on the cluster and chose namespace openshift-storage. Click Subscribe.

Figure 5. OCP Subscribe to OpenShift Container Storage
Now you can go back to your terminal window to check the progress of the installation.

watch oc -n openshift-storage get csv

Example output:

NAME DISPLAY VERSION REPLACES PHASE

ocs-operator.v0.0.284 OpenShift Container Storage 0.0.284 Succeeded

You can exit by pressing Ctrl+C
The resource csv is a shortened word for clusterserviceversions.operators.coreos.com.

> Caution: Please wait until the operator PHASE changes to Succeeded. This will mark that the installation of your operator was successful. Reaching this state can take several minutes.
You will now also see some new operator pods in the new openshift-storage namespace:

oc -n openshift-storage get pods

Example output:

NAME READY STATUS RESTARTS AGE

noobaa-operator-7c55776bf9-kbcjp 1/1 Running 0 3m16s

ocs-operator-967957d84-9lc76 1/1 Running 0 3m16s

rook-ceph-operator-8444cfdc4c-9jm8p 1/1 Running 0 3m16s

Now switch back to your Openshift Web Console for the remainder of the installation for OCS 4.
Navigate to the Operators menu on the left and select Installed Operators. Make sure the selected project is set to openshift-storage. What you see, should be similar to the following example picture:

Figure 6. Installed operators: 1) Make sure you are in the right project; 2) Check Operator status; 3) Click on Openshift Container Storage Operator
Click on Openshift Container Storage Operator to get to the OCS configuration screen.

Figure 7. OCS configuration screen

On the top of the OCS configuration screen, scroll over to Storage cluster and click on Create OCS Cluster Service. If you do not see Create OCS Cluster Service refresh your browser window.

Figure 8. OCS Create Storage Cluster

A dialog box will come up next.

Figure 9. OCS create a new storage cluster

> Caution: Make sure to select three workers in different availability zones using instructions below.

To select the appropriate worker nodes of your OCP 4 cluster you can find them by searching for the node label role=storage-node.

oc get nodes --show-labels | grep storage-node |cut -d' ' -f1

Select the three nodes that resulted from the command above. Then click on the button Create below the dialog box where you selected the 3 workers with a checkmark.

> Note: If your worker nodes do not have the label role=storage-node just select 3 worker nodes that meet requirements for OCS nodes (16 vCPUs and 64 GB Memory) and in different availability zones. It would be a good practice to add a unique label to OCP nodes that are to be used for creating the Storage Cluster prior to this step so they are easy to find in list of OCP nodes. In this case it was done by adding this label, role=storage-node, in the machineset YAML files that you used earlier to create the new OCS worker nodes.
In the background this will start initiating a lot of new pods in the openshift-storage namespace, as can be seen on the CLI:

oc -n openshift-storage get pods

Example of a in process installation of the OCS storage cluster:

NAME READY STATUS RESTARTS AGE

csi-cephfsplugin-72n5r 3/3 Running 0 52s

csi-cephfsplugin-cgc4p 3/3 Running 0 52s

csi-cephfsplugin-ksp9j 3/3 Running 0 52s

csi-cephfsplugin-provisioner-849895689c-5mcvm 4/4 Running 0 52s

csi-cephfsplugin-provisioner-849895689c-k784q 4/4 Running 0 52s

csi-cephfsplugin-sfwwg 3/3 Running 0 52s

csi-cephfsplugin-vmv77 3/3 Running 0 52s

csi-rbdplugin-56pwz 3/3 Running 0 52s

csi-rbdplugin-9cwwt 3/3 Running 0 52s

csi-rbdplugin-pmw5g 3/3 Running 0 52s

csi-rbdplugin-provisioner-58d79d7895-69vx9 4/4 Running 0 52s

csi-rbdplugin-provisioner-58d79d7895-mkr78 4/4 Running 0 52s

csi-rbdplugin-pvn82 3/3 Running 0 52s

csi-rbdplugin-zdz5c 3/3 Running 0 52s

noobaa-operator-7ffd9dc86-nmfwm 1/1 Running 0 40m

ocs-operator-9694fd887-mwmsn 0/1 Running 0 40m

rook-ceph-detect-version-544tg 0/1 Terminating 0 46s

rook-ceph-mon-a-canary-6874bdb7-rjv95 0/1 ContainerCreating 0 14s

rook-ceph-mon-b-canary-5d5b47ccfd-wpvnp 0/1 ContainerCreating 0 8s

rook-ceph-mon-c-canary-56969776fc-xgkvw 0/1 ContainerCreating 0 3s

rook-ceph-operator-5dc5f9d7fb-zd7qs 1/1 Running 0 40m

You can also watch the deployment using the Openshift Web Console by going back to the Openshift Container Storage Operator screen and selecting All instances.
Please wait until all Pods are marked as Running in the CLI or until you see all instances shown below as ReadyStatus in the Web Console. Some instances may stay in Unknown Status which is not a concern if your Ready status matches the following diagram:

Figure 10. OCS instance overview after cluster install is finished

oc -n openshift-storage get pods

Output when the cluster installation is finished

NAME READY STATUS RESTARTS AGE

csi-cephfsplugin-72n5r 3/3 Running 0 10m

csi-cephfsplugin-cgc4p 3/3 Running 0 10m

csi-cephfsplugin-ksp9j 3/3 Running 0 10m

csi-cephfsplugin-provisioner-849895689c-5mcvm 4/4 Running 0 10m

csi-cephfsplugin-provisioner-849895689c-k784q 4/4 Running 0 10m

csi-cephfsplugin-sfwwg 3/3 Running 0 10m

csi-cephfsplugin-vmv77 3/3 Running 0 10m

csi-rbdplugin-56pwz 3/3 Running 0 10m

csi-rbdplugin-9cwwt 3/3 Running 0 10m

csi-rbdplugin-pmw5g 3/3 Running 0 10m

csi-rbdplugin-provisioner-58d79d7895-69vx9 4/4 Running 0 10m

csi-rbdplugin-provisioner-58d79d7895-mkr78 4/4 Running 0 10m

csi-rbdplugin-pvn82 3/3 Running 0 10m

csi-rbdplugin-zdz5c 3/3 Running 0 10m

noobaa-core-0 2/2 Running 0 6m3s

noobaa-operator-7ffd9dc86-nmfwm 1/1 Running 0 49m

ocs-operator-9694fd887-mwmsn 1/1 Running 0 49m

rook-ceph-drain-canary-ip-10-0-136-247.ec2.internal-55c658klgqg 1/1 Running 0 6m10s

rook-ceph-drain-canary-ip-10-0-157-178.ec2.internal-758658dr4jw 1/1 Running 0 6m26s

rook-ceph-drain-canary-ip-10-0-168-170.ec2.internal-5b499cfc6xl 1/1 Running 0 6m27s

rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-8568c68dmzctp 1/1 Running 0 5m57s

rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-77b78d-6jhcw 1/1 Running 0 5m57s

rook-ceph-mgr-a-7767f6cf56-2s6mt 1/1 Running 0 7m24s

rook-ceph-mon-a-65b6ffb7f4-57gds 1/1 Running 0 8m50s

rook-ceph-mon-b-6698bf6d5-zml6j 1/1 Running 0 8m25s

rook-ceph-mon-c-55c8f47456-7x455 1/1 Running 0 7m54s

rook-ceph-operator-5dc5f9d7fb-zd7qs 1/1 Running 0 49m

rook-ceph-osd-0-7fc4dd559b-kgvgb 1/1 Running 0 6m27s

rook-ceph-osd-1-9d9dc8f4b-kh8qr 1/1 Running 0 6m27s

rook-ceph-osd-2-559fb96fcb-zc97d 1/1 Running 0 6m10s

rook-ceph-osd-prepare-ocs-deviceset-0-0-g9j2d-wvqj5 0/1 Completed 0 7m2s

rook-ceph-osd-prepare-ocs-deviceset-1-0-h59x8-l5wjs 0/1 Completed 0 7m2s

rook-ceph-osd-prepare-ocs-deviceset-2-0-74spm-tdlb6 0/1 Completed 0 7m1s

2.3. Getting to know the Storage Dashboards

You can now also check the status of your storage cluster with the OCS specific Dashboards that are included in your Openshift Web Console. You can reach this by clicking on Home on your left navigation bar, then selecting Dashboards and finally clicking on Persistent Storage on the top navigation bar of the content page.

> Note: If you just finished your OCS 4 deployment it could take 5-10 minutes for your Dashboards to fully populate.

Figure 11. OCS Dashboard after successful backing storage installation

1 | Health | Quick overview of the general health of the storage cluster
2 | Details | Overview of the deployed storage cluster version and backend provider
3 | Inventory | List of all the resources that are used and offered by the storage system
4 | Events | Live overview of all the changes that are being done affecting the storage cluster
5 | Utilization | Overview of the storage cluster usage and performance
OCS ships with a Dashboard for the Object Store service as well. From within the Dashboard menu click on the Object Service on the top navigation bar of the content page.

Figure 12. OCS Multi-Cloud-Gateway Dashboard after successful installation

1 | Health | Quick overview of the general health of the Multi-Cloud-Gateway
2 | Details | Overview of the deployed MCG version and backend provider including a link to the MCG Dashboard
3 | Buckets | List of all the ObjectBucket with are offered and ObjectBucketClaims which are connected to them
4 | Resource Providers | Shows the list of configured Resource Providers that are available as backing storage in the MCG

Once this is all healthy, you will be able to use the three new StorageClasses created during the OCS 4 Install:
ocs-storagecluster-ceph-rbd
ocs-storagecluster-cephfs
openshift-storage.noobaa.io
You can see these three StorageClasses from the Openshift Web Console by expanding the Storage menu in the left navigation bar and selecting Storage Classes. You can also run the command below:

oc -n openshift-storage get sc

Please make sure the three storage classes are available in your cluster before proceeding.

> Note: The NooBaa pod used the ocs-storagecluster-ceph-rbd storage class for creating a PVC for mounting to it’s db container.

2.4. Using the Rook-Ceph toolbox to check on the Ceph backing storage

Since the Rook-Ceph toolbox is not shipped with OCS, we need to deploy it manually. For this, we can leverage the upstream toolbox.yaml file, but we need to modify the namespace as shown below.

curl -s https://raw.githubusercontent.com/rook/rook/release-1.1/cluster/examples/kubernetes/ceph/toolbox.yaml | sed 's/namespace: rook-ceph/namespace: openshift-storage/g'| oc apply -f -

After the rook-ceph-tools Pod is Running you can access the toolbox like this:

TOOLS_POD=$(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name)

oc rsh -n openshift-storage $TOOLS_POD

Once inside the toolbox, try out the following Ceph commands:
ceph status
ceph osd status
ceph osd tree
ceph df
rados df
ceph versions
Example output:

sh-4.2# ceph status

cluster:

id: 786dbab2-ae4f-4352-8d83-5e27c6a4f341

health: HEALTH_OK
services:
 mon: 3 daemons, quorum a,b,c (age 105m)
 mgr: a(active, since 104m)
 mds: ocs-storagecluster-cephfilesystem:1 {0=ocs-storagecluster-cephfilesystem-a=up:active} 1 up:standby-replay
 osd: 3 osds: 3 up (since 104m), 3 in (since 104m)
data:
 pools: 3 pools, 24 pgs
 objects: 100 objects, 114 MiB
 usage: 3.2 GiB used, 3.0 TiB / 3.0 TiB avail
 pgs: 24 active+clean

io:
 client: 1.2 KiB/s rd, 39 KiB/s wr, 2 op/s rd, 3 op/s wr
 

You can exit the toolbox by either pressing Ctrl+D or by executing exit.

Resources and Feedback

To find out more about OpenShift Container Storage or to take a test drive, visit https://www.openshift.com/products/container-storage/.

If you would like to learn more about what the OpenShift Container Storage team is up to or provide feedback on any of the new 4.2 features, take this brief 3-minute survey.