OpenShift Container Platform 3.6: Streamlined installation and configuration of Red Hat Gluster Storage for containers

By Erin Boyd, Jose Rivera, and Scott Creeley, Red Hat

Did you ever get a new, exciting toy only to have that excitement squashed by the phrase "Batteries not included"?

With the introduction of Red Hat OpenShift Container Platform 3.6, no longer will customers have to wait or jump extra hurdles to get resilient, persistent storage with their new installations. Now they can more easily deploy Red Hat Gluster Storage ready for use by their containerized applications—This is PaaS with batteries included!

With the release of Red Hat OpenShift Container Platform 3.6, users will have the convenience of using a single tool to use Red Hat Gluster Storage as either container-native storage (CNS) or container-ready storage (CRS) alongside the rest of their OpenShift installations. As part of the OpenShift Advanced Installation, users can specify two new storage options: Red Hat Gluster Storage for (1) hosted registry storage or (2) general application storage. To facilitate evaluation of these, an Openshift Container Platform evaluation subscription now includes Red Hat Gluster Storage evaluation binaries and subscriptions.

Following is a sample inventory file that would be used with an OpenShift Container Platform Advanced Installation that deploys two CNS clusters for both hosted registry storage and general application storage.

[OSEv3:children]
 masters
 nodes
 glusterfs_registry
 glusterfs

[OSEv3:vars]
 ansible_ssh_user=root
 openshift_master_default_subdomain=cloudapps.example.com
 openshift_deployment_type=openshift-enterprise
 openshift_hosted_registry_storage_kind=glusterfs
 openshift_disable_check=disk_availability,memory_availability

[nodes]
 master1 node=True storage=True master=True openshift_schedulable=False
 node1 node=True storage=True openshift_node_labels="{'region': 'infra'}"
 openshift_schedulable=True
 node2 node=True storage=True openshift_node_labels="{'region': 'infra'}"
 openshift_schedulable=True
 node3 node=True storage=True openshift_node_labels="{'region': 'infra'}"
 openshift_schedulable=True
 node4 node=True storage=True openshift_schedulable=True
 node5 node=True storage=True openshift_schedulable=True
 node6 node=True storage=True openshift_schedulable=True

[glusterfs_registry]
 node1 glusterfs_devices="[ '/dev/xvdc' ]"
 node2 glusterfs_devices="[ '/dev/xvdc' ]"
 node3 glusterfs_devices="[ '/dev/xvdc' ]"

[glusterfs]
 node4 glusterfs_devices="[ '/dev/xvdc' ]"
 node5 glusterfs_devices="[ '/dev/xvdc' ]"
 node6 glusterfs_devices="[ '/dev/xvdc' ]"

[masters]
 master1 node=True storage=True master=True openshift_schedulable=False

Let's go over the highlighted portions in detail.

The first section defines the host groups the installation will be using. We've defined two new groups: (1) glusterfs_registry and (2) glusterfs. The first specifies a cluster that will host a single volume for use exclusively by a hosted registry. The second specifies a cluster for general application storage and will, by default, come with a Storage Class to enable dynamic provisioning.

[OSEv3:children]
 masters
 nodes
 glusterfs_registry
 glusterfs

In the following section, we indicate that we want the hosted registry to use Red Hat Gluster Storage for its storage needs.

[OSEv3:vars]
 ansible_ssh_user=root
 openshift_master_default_subdomain=cloudapps.example.com
 openshift_deployment_type=openshift-enterprise
 openshift_hosted_registry_storage_kind=glusterfs
 openshift_disable_check=disk_availability,memory_availability

In the [nodes] section, we need to specify all nodes in the OpenShift Container Platform cluster. For our installation, we also need to specify which nodes will run pods for the hosted registry. This is done by specifying "openshift_node_labels="{'region': 'infra'}"" for each such node. It is recommended to have at least three nodes running your hosted registry.

[nodes]
 master1 node=True storage=True master=True openshift_schedulable=False
 node1 node=True storage=True openshift_node_labels="{'region': 'infra'}"
 openshift_schedulable=True
 node2 node=True storage=True openshift_node_labels="{'region': 'infra'}"
 openshift_schedulable=True
 node3 node=True storage=True openshift_node_labels="{'region': 'infra'}"
 openshift_schedulable=True
 node4 node=True storage=True openshift_schedulable=True
 node5 node=True storage=True openshift_schedulable=True
 node6 node=True storage=True openshift_schedulable=True

Now we get to our new sections where we specify the nodes that will be used for storage. CNS and CRS require that each cluster have a minimum of three nodes. Multiple clusters can not share a given node. Because we are deploying two clusters, we need to specify six nodes total. It is also required that each node have at least one dedicated, bare storage device (no data or formatting of any kind) for exclusive use by Red Hat Gluster Storage.

Our first new section is [glusterfs_registry]. Here we specify the nodes of the Red Hat Gluster Storage cluster and the storage devices on those nodes that will be used for a hosted registry's storage. It is not required that these nodes be the same as the ones running the hosted registry.

[glusterfs_registry]
 node1 glusterfs_devices="[ '/dev/xvdc' ]"
 node2 glusterfs_devices="[ '/dev/xvdc' ]"
 node3 glusterfs_devices="[ '/dev/xvdc' ]"

Our second new section, [glusterfs], is used for specifying the Red Hat Gluster Storage cluster and storage devices that will be used for general application storage. These storage devices must also be for exclusive use by Red Hat Gluster Storage. As mentioned, these nodes may not also be part of the cluster used by [glusterfs_registry]. In the case of CNS, it is not required that these nodes be dedicated exclusively to serving storage; CNS pods can coexist with other application pods.

[glusterfs]
 node4 glusterfs_devices="[ '/dev/xvdc' ]"
 node5 glusterfs_devices="[ '/dev/xvdc' ]"
 node6 glusterfs_devices="[ '/dev/xvdc' ]"

Once the installer is complete, the user can see the pre-defined Storage Class by executing:

# oc get storageclasses
 NAME TYPE
 glusterfs-storage kubernetes.io/glusterfs

This Storage Class can be used for applications by specifying a Persistent Volume Claim to dynamically provision the required storage volume:

apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
 name: mypvc
 namespace: glusterfs
 spec:
 accessModes:
 - ReadWriteOnce
 resources:
 requests:
 storage: 100Gi
 storageClassName: glusterfs-storage

And that's it, your PaaS solution with built-in storage is ready to go! If you want to tune the installation further, more options are available in the Advanced Installation, and a demo video is available here.