Gluster for OpenShift - Part 1: Container-Ready Storage

OpenShift Container Platform (OCP) offers many different types of persistent storage. Persistent storage ensures that data that should persist between builds and container migrations continues to be available. When choosing a persistent storage backend ensure that the backend supports the scaling, speed, dynamic provisioning, RWX/RWO support and redundancy that the project requires. Container-Ready Storage (CRS), or native Gluster for OCP, is defined by the concept of persistent volumes which are OCP created objects that allow for storage to be defined and then used by pods to allow for data persistence.

Requesting of persistent volumes (PV) is done by using a persistent volume claim (PVC). This claim, when successfully fulfilled by the system will also mount the persistent storage to a specific directory within a pod or multiple pods. This directory is referred to as the mountPath and facilitated using a concept known as bind-mount.

The OpenShift ansible contrib repo provides reference architectures for many platform providers including: AWS, Azure, GCE, Openstack, RHEV and VMware.

The github repo with playbooks and scripts to deploy OpenShift on VMware as well CRS are located here:

https://github.com/openshift/openshift-ansible-contrib/tree/master/reference-architecture/vmware-ansible.

These playbooks and scripts will guide you from start to finish to deploying OCP on VMware vCenter utilizing Container-Ready Storage.

Deploying Container-Ready Storage

A python script named add-node.py is provided in the openshift-ansible-contrib git repository. When add-node.py is used with the --node_type=storage option the following will be completely automated (dependent on variable "container_storage=cns" in the ocp-on-vmware.ini file).

1. Create three VMware virtual machines with 32 GB Mem and 2 vCPUs
2.  Register the new machines with Red Hat
3. Install the prerequisites for CRS for Gluster on each machine
4. Add a VMDK volume to each node as an available block device to be used for CRS
5. Create a heketi topology.json file using virtual machine hostnames and new VMDK device name
6. Install heketi and heketi-cli packages on one of the CRS nodes
7. Copy heketi public key to all CRS nodes
8. Modify heketi.json file with user supplied admin and user paswords and other necessary configuration for passwordless SSH to all CRS nodes
9. Using heketi-cli and topology.json file deploy the new CRS cluster
10. Create heketi-secret and new StorageClass object for PVC creation

Here is example of what is automated for step 9 above. Loading the CRS topology.json file to create a new CRS Trusted Storage Pool (TSP). This is done from the CRS node where heketi was deployed. The topology.json file is archived on this node for future modification to add more storage devices, more storage nodes, etc.

[code language="bash"]
$ cat topology.json
{
"clusters": [
{
"nodes": [
{
"devices": [
"/dev/sdd"
],
"node": {
"hostnames": {
"manage": [
"ocp3-crs0.dpl.local"
],
"storage": [
"172.0.10.215"
]
},
"zone": 1
}
},
{
"devices": [
"/dev/sdd"
],
"node": {
"hostnames": {
"manage": [
"ocp3-crs1.dpl.local"
],
"storage": [
"172.0.10.216"
]
},
"zone": 2
}
},
{
"devices": [
"/dev/sdd"
],
"node": {
"hostnames": {
"manage": [
"ocp3-crs2.dpl.local"
],
"storage": [
"172.0.10.217"
]
},
"zone": 3
}
}
]
}
]
}
[/code]

Now export the heketi environment values and load the topology.json file:

[code language="bash"]
$ export HEKETI_CLI_SERVER=http://ocp3-crs-0.dpl.local:8080
$ export HEKETI_CLI_USER=admin
$ export HEKETI_CLI_KEY=myS3cr3tpassw0rd
$ heketi-cli topology load --json=topology.json
Creating cluster ... ID: bb802020a9c2c5df45f42075412c8c05
Creating node ocp3-crs-0.dpl.local ... ID: b45d38a349218b8a0bab7123e004264b
Adding device /dev/sdd ... OK
Creating node ocp3-crs-1.dpl.local ... ID: 2b3b30efdbc3855a115d7eb8fdc800fe
Adding device /dev/sdd ... OK
Creating node ocp3-crs-2.dpl.local ... ID: c7d366ae7bd61f4b69de7143873e8999
Adding device /dev/sdd ... OK
[/code]

Creating Heketi secret and CRS StorageClass OCP objects

For step 10 above, OCP allows for the use of secrets so that items do not need to be stored in clear text. The admin password for heketi, specified during configuration of the heketi.json file, should be stored in base64-encoding. OCP can refer to this secret instead of specifying the password in clear text.

[code language="bash"]
$ echo -n myS3cr3tpassw0rd | base64
bXlTM2NyM3RwYXNzdzByZA==
[/code]
On the master or workstation with the OCP client installed with cluster-admin privileges use the base64 password string in the following YAML to define the secret in OCP’s default namespace.

[code language="bash"]
$ cat heketi-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: heketi-secret
namespace: default
data:
key: bXlTM2NyM3RwYXNzdzByZA==
type: kubernetes.io/glusterfs
[/code]
Create the secret by using the following OCP CLI command.

[code language="bash"]
$ oc create -f heketi-secret.yaml
secret "heketi-secret" created
[/code]
A StorageClass object requires certain parameters to be defined to successfully create the resource. Use the values of the exported environment variables from the previous steps to define the resturl, restuser, secretNamespace, and secretName. The key benefit of using a StorageClass object is that the persistent storage can be created with access modes ReadWriteOnce(RWO), ReadOnlyMany(ROX), or ReadWriteMany(RWX).

[code language="bash"]
$ cat storageclass.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
name: crs-gluster
provisioner: kubernetes.io/glusterfs
parameters:
resturl: "http://ocp3-crs-0.dpl.local:8080"
restauthenabled: "true"
restuser: "admin"
secretNamespace: "default"
secretName: "heketi-secret"
[/code]

Once the StorageClass yaml file has been created, use the oc create command to create the object in OpenShift.

[code language="bash"]
$ oc create -f storageclass.yaml
[/code]
To validate the StorageClass object was created perform the following:

[code language="bash"]
$ oc get storageclass
NAME TYPE
crs-gluster kubernetes.io/glusterfs

$ oc describe storageclass crs-gluster
Name:crs-gluster
IsDefaultClass:No
Annotations:
Provisioner:kubernetes.io/glusterfs
Parameters:restauthenabled=true,resturl=http://ocp3-crs-0.dpl.local:8080,restuser=admin,secretName=heketi-secret,secretNamespace=default
No events.
[/code]

Creating a Dynamic Persistent Volume Claim (PVC)

The Storage Class created in the previous section allows for storage to be dynamically provisioned using the CRS resources. The example below shows a dynamically provisioned gluster volume being requested from the crs-gluster StorageClass object. A sample persistent volume claim is provided below:

[code language="bash"]
$ vi db-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: db
annotations:
volume.beta.kubernetes.io/storage-class: crs-gluster
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi

$ oc create -f db-claim.yaml
persistentvolumeclaim "db" created
[/code]

Dynamic PV claims can also be made by configuring OpenShift templates with the desired StorageClass object name. Example show below for how to modify the default openshift mysql-persistent template file. If the StorageClass object name is not specified the default storageclass will be used if one is used. Also, the default size for the PVC is 1GiB so make sure to increase this size if larger size is needed.

[code language="bash"]
$ oc export template/mysql-persistent -n openshift -o yaml > mysql-persistent.yaml
$ cat mysql-persistent.yaml
....omitted....
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ${DATABASE_SERVICE_NAME}
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: ${VOLUME_CAPACITY}
....omitted....
[/code]
Modify template with desired StorageClass object name to create dynamic PVC or gluster volume for mount-path=/var/lib/mysql/data.

[code language="bash"]
$ vim mysql-persistent.yaml
....omitted....
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ${DATABASE_SERVICE_NAME}
annotations:
volume.beta.kubernetes.io/storage-class: crs-gluster
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: ${VOLUME_CAPACITY}
....omitted....
[/code]

... Gluster for OpenShift - Part 2: Container-Native Storage coming soon!

DEPLOYING A RED HAT OPENSHIFT CONTAINER PLATFORM 3 ON VMWARE VCENTER 6 UTILIZING GLUSTER CONTAINER-NATIVE STORAGE This reference architecture describes how to deploy and manage Red Hat OpenShift Container Platform on VMware vCenter utilizing Gluster for persistent storage.