Contact

ByAnnette Clewett anJose A. Rivera

With the release of Red Hat OpenShift Container Platform 3.10, we've officially rebranded what used to be referred to as Red Hat Container-Native Storage (CNS) as Red Hat OpenShift Container Storage(OCS). Versioning remains sequential (i.e, OCS version 3.10 is the follow on to CNS 3.9). You'll continue to have the convenience of OCS 3.10 as part of the normal OpenShift deployment process in a single step, and OpenShift Container Platform (OCP) evaluation subscription has access to OCS evaluation binaries and subscriptions.

OCS 3.10 introduces an important feature for container-based storage with OpenShift. Arbiter volume support allows for there to be only two replica copies of the data, while still providing split-brain protection and ~30% savings in storage infrastructure versus a replica-3 volume. This release also hardens block support for backing OpenShift infrastructure services. Detailed information on the value and use of OCS 3.10 features can be found here.

OCS 3.10 installation with OCP 3.10 Advanced Installer

Let’s now take a look at the installation of OCS with the OCP Advanced Installer. OCS can provide persistent storage for both OCP's infrastructure applications (e.g., integrated registry, logging, and metrics), as well as  general application data consumption. Typically, both options are used in parallel, resulting in two separate OCS clusters being deployed in a single OCP environment. It’s also possible to use a single OCS cluster for both purposes.

Following is an example of a partialinventory file with selected options concerning deployment of OCS for applications and an additional OCS cluster for infrastructure workloads like registry, logging, and metrics storage. When using these options for your deployment, values with specific sizes (e.g., openshift_hosted_registry_storage_volume_size=10Gi) or node selectors  (e.g., node-role.kubernetes.io/infra=true) should be adjusted for your particular deployment needs.

If you're planning to use gluster-block volumes for logging and metrics, they can now be installed when OCP is installed. (Of course, they can also be installed later.)

[OSEv3:children]...nodesglusterfsglusterfs_registry[OSEv3:vars]...     # registryopenshift_hosted_registry_storage_kind=glusterfs      openshift_hosted_registry_storage_volume_size=10Gi  openshift_hosted_registry_selector="node-role.kubernetes.io/infra=true"# loggingopenshift_logging_install_logging=trueopenshift_logging_es_pvc_dynamic=trueopenshift_logging_es_pvc_size=50Giopenshift_logging_es_cluster_size=3openshift_logging_es_pvc_storage_class_name='glusterfs-registry-block'openshift_logging_kibana_nodeselector={"node-role.kubernetes.io/infra": "true"}openshift_logging_curator_nodeselector={"node-role.kubernetes.io/infra": "true"}openshift_logging_es_nodeselector={"node-role.kubernetes.io/infra": "true"}# metricsopenshift_metrics_install_metrics=trueopenshift_metrics_storage_kind=dynamicopenshift_metrics_storage_volume_size=20Giopenshift_metrics_cassandra_pvc_storage_class_name='glusterfs-registry-block'openshift_metrics_hawkular_nodeselector={"node-role.kubernetes.io/infra": "true"}openshift_metrics_cassandra_nodeselector={"node-role.kubernetes.io/infra": "true"}openshift_metrics_heapster_nodeselector={"node-role.kubernetes.io/infra": "true"}# Container image to use for glusterfs podsopenshift_storage_glusterfs_image="registry.access.redhat.com/rhgs3/rhgs-server-rhel7:v3.10"# Container image to use for gluster-block-provisioner podopenshift_storage_glusterfs_block_image="registry.access.redhat.com/rhgs3/rhgs-gluster-block-prov-rhel7:v3.10"# Container image to use for heketi podsopenshift_storage_glusterfs_heketi_image="registry.access.redhat.com/rhgs3/rhgs-volmanager-rhel7:v3.10"# OCS storage cluster for applicationsopenshift_storage_glusterfs_namespace=app-storageopenshift_storage_glusterfs_storageclass=trueopenshift_storage_glusterfs_storageclass_default=falseopenshift_storage_glusterfs_block_deploy=false   # OCS storage cluster for OpenShift infrastructureopenshift_storage_glusterfs_registry_namespace=infra-storage openshift_storage_glusterfs_registry_storageclass=false      openshift_storage_glusterfs_registry_block_deploy=true  openshift_storage_glusterfs_registry_block_host_vol_create=true    openshift_storage_glusterfs_registry_block_host_vol_size=200  openshift_storage_glusterfs_registry_block_storageclass=trueopenshift_storage_glusterfs_registry_block_storageclass_default=false...
[nodes]ose-app-node01.ocpgluster.com openshift_node_group_name="node-config-compute"ose-app-node02.ocpgluster.com openshift_node_group_name="node-config-compute"ose-app-node03.ocpgluster.com openshift_node_group_name="node-config-compute"ose-app-node04.ocpgluster.com openshift_node_group_name="node-config-compute"ose-infra-node01.ocpgluster.com openshift_node_group_name="node-config-infra"ose-infra-node02.ocpgluster.com openshift_node_group_name="node-config-infra"ose-infra-node03.ocpgluster.com openshift_node_group_name="node-config-infra"[glusterfs]ose-app-node01.ocpgluster.com glusterfs_zone=1 glusterfs_devices='[ "/dev/xvdf" ]'   ose-app-node02.ocpgluster.com glusterfs_zone=2 glusterfs_devices='[ "/dev/xvdf" ]'ose-app-node03.ocpgluster.com glusterfs_zone=3 glusterfs_devices='[ "/dev/xvdf" ]'ose-app-node04.ocpgluster.com glusterfs_zone=1 glusterfs_devices='[ "/dev/xvdf" ]'[glusterfs_registry]ose-infra-node01.ocpgluster.com glusterfs_zone=1 glusterfs_devices='[ "/dev/xvdf" ]'ose-infra-node02.ocpgluster.com glusterfs_zone=2 glusterfs_devices='[ "/dev/xvdf" ]'ose-infra-node03.ocpgluster.com glusterfs_zone=3 glusterfs_devices='[ "/dev/xvdf" ]'

Inventory file options explained

The first section of the inventory file defines the host groups the installation will be using. We’ve defined two new groups: (1) glusterfs and (2) glusterfs_registry. The settings for either group all start with either openshift_storage_glusterfs_ or openshift_storage_glusterfs_registry. In each group, the nodes that will make up the OCS cluster are listed, and the devices ready for exclusive use by OCS are specified (glusterfs_devices=).

The first group of hosts in glusterfs specifies a cluster for general-purpose application storage and will, by default, come with the StorageClass glusterfs-storage to enable dynamic provisioning. For high availability of storage, it's very important to have four nodes for the general-purpose application cluster, glusterfs.

The second group, glusterfs_registry, specifies a cluster that will host a single, statically deployed PersistentVolume for use exclusively by a hosted registry that can scale. This cluster will not offer a StorageClass for file-based PersistentVolumes with the options and values as they are currently configured (openshift_storage_glusterfs_registry_storageclass=false). This cluster will also support gluster-block (openshift_storage_glusterfs_registry_block_deploy=true). PersistentVolume creation can be done via StorageClass glusterfs-registry-block (openshift_storage_glusterfs_registry_block_storageclass=true). Special attention should be given to choosing the size for openshift_storage_glusterfs_registry_block_host_vol_size. This is the hosting volume for gluster-block devices that will be created for logging and metrics. Make sure that the size can accommodate all these block volumes and that you have sufficient storage if another hosting volume must be created.

If you want to tune the installation, more options are available in the Advanced Installation. To automate the generation of required inventory file options as shown previously, check out this newly available red-hat-storage tool called “CNS Inventory file Creator” or CIC (alpha version at this time). The CIC tool creates CNS or OCS inventory file options for both OCP 3.9 and OCP 3.10, respectively. CIC will ask a series of questions about the OpenShift hosts, the storage devices, sizes of PersistentVolumes for registry, logging and metrics and has baked-in checks to make sure the OCP installation will be successful. This tool  is currently alpha state, and we're looking for feedback. Download it from github repository openshift-cic.

Single OCS cluster installation

Again, it is possible to support both general-application storage and infrastructure storage in a single OCS cluster. To do this, the inventory file options will change slightly for logging and metrics. This is because when there is only one cluster, the gluster-block StorageClass would be glusterfs-storage-block. The registry PV will be created on this single cluster if the second cluster, [glusterfs_registry], does not exist. For high availability, it's very important to have four nodes for this cluster.  Also, special attention should be given to choosing the size for openshift_storage_glusterfs_block_host_vol_size. This is the hosting volume for gluster-block devices that will be created for logging and metrics. Make sure that the size can accommodate all these block volumes and that you have sufficient storage if another hosting volume must be created.

[OSEv3:children]
...nodesglusterfs[OSEv3:vars]...     # registry...# loggingopenshift_logging_install_logging=true...openshift_logging_es_pvc_storage_class_name='glusterfs-storage-block'... # metricsopenshift_metrics_install_metrics=true...openshift_metrics_cassandra_pvc_storage_class_name='glusterfs-storage-block'...# OCS storage cluster for applicationsopenshift_storage_glusterfs_namespace=app-storageopenshift_storage_glusterfs_storageclass=trueopenshift_storage_glusterfs_storageclass_default=falseopenshift_storage_glusterfs_block_deploy=trueopenshift_storage_glusterfs_block_host_vol_create=trueopenshift_storage_glusterfs_block_host_vol_size=100openshift_storage_glusterfs_block_storageclass=trueopenshift_storage_glusterfs_block_storageclass_default=false...[nodes]ose-app-node01.ocpgluster.com openshift_node_group_name="node-config-compute"   ose-app-node02.ocpgluster.com openshift_node_group_name="node-config-compute" ose-app-node03.ocpgluster.com openshift_node_group_name="node-config-compute" ose-app-node04.ocpgluster.com openshift_node_group_name="node-config-compute" [glusterfs]ose-app-node01.ocpgluster.com glusterfs_zone=1 glusterfs_devices='[ "/dev/xvdf" ]'   ose-app-node02.ocpgluster.com glusterfs_zone=2 glusterfs_devices='[ "/dev/xvdf" ]'ose-app-node03.ocpgluster.com glusterfs_zone=3 glusterfs_devices='[ "/dev/xvdf" ]'ose-app-node04.ocpgluster.com glusterfs_zone=1 glusterfs_devices='[ "/dev/xvdf" ]'

OCS 3.10 uninstall

With the OCS 3.10 release, the uninstall.yml playbook can be used to remove all gluster and heketi resources. This might come in handy when there are errors in inventory file options that cause the gluster cluster to deploy incorrectly.

If you're removing an OCS installation that is currently being used by any applications, you should remove those applications before removing OCS, because they will lose access to storage. This includes infrastructure applications like registry, logging, and metrics that have PV claims created using the glusterfs-storage and glusterfs-storage-block Storage Class resources.

You can remove logging and metrics resources by re-running the deployment playbooks like this:

ansible-playbook -i <path_to_inventory_file> -e
"openshift_logging_install_logging=false"
/usr/share/ansible/openshift-ansible/playbooks/openshift-logging/config.ymlansible-playbook -i <path_to_inventory_file> -e
"openshift_logging_install_metrics=false"
/usr/share/ansible/openshift-ansible/playbooks/openshift-metrics/config.yml

Make sure to manually remove any logging or metrics PersistentVolumeClaims. The associated PersistentVolumes will be deleted automatically.

If you have the registry using a glusterfs PersistentVolume, remove it with the following command:

oc delete deploymentconfig docker-registryoc delete pvc registry-claimoc delete pv registry-volumeoc delete service glusterfs-registry-endpoints

If running the uninstall.yml because a deployment failed, run the uninstall.yml playbook with the following variables to wipe the storage devices for both glusterfs and glusterfs_registry before trying the OCS installation again.

ansible-playbook -i <path_to_inventory file> -e
"openshift_storage_glusterfs_wipe=True" -e
"openshift_storage_glusterfs_registry_wipe=true"
/usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/uninstall.yml

OCS 3.10 post installation for applications, registry, logging and metrics

You can add OCS clusters and resources to an existing OCP install using the following command. This same process can be used if OCS has been uninstalled due to errors.

ansible-playbook -i <path_to_inventory_file>
/usr/share/ansible/openshift-ansible/playbooks/openshift-glusterfs/config.yml

After the new cluster(s) is created and validated, you can deploy the registry using a newly created glusterfs ReadWriteMany volume. Run this playbook to create the registry resources:

ansible-playbook -i <path_to_inventory_file>
/usr/share/ansible/openshift-ansible/playbooks/openshift-hosted/config.yml

You can now deploy logging and metrics resources by re-running these deployment playbooks:

ansible-playbook -i <path_to_inventory_file>
/usr/share/ansible/openshift-ansible/playbooks/openshift-logging/config.ymlansible-playbook -i <path_to_inventory_file>
/usr/share/ansible/openshift-ansible/playbooks/openshift-metrics/config.yml

Want to learn more?

For hands-on experience combining OpenShift and OCS, check out our test drive, a free, in-browser lab experience that walks you through using both. Also, watch this short video explaining why to use OCS with OCP. Detailed information on the value and use of OCS 3.10 features can be found here.


Red Hat logoLinkedInYouTubeFacebookTwitter

Produits

Outils

Essayer, acheter et vendre

Communication

À propos de Red Hat

Premier éditeur mondial de solutions Open Source pour les entreprises, nous fournissons des technologies Linux, cloud, de conteneurs et Kubernetes. Nous proposons des solutions stables qui aident les entreprises à jongler avec les divers environnements et plateformes, du cœur du datacenter à la périphérie du réseau.

Inscrivez-vous à la newsletter Red Hat Shares.

S'inscrire maintenant

Sélectionner une langue