Subscribe to the feed

Following the “OpenShift on OpenStack: Availability Zones” series, in the first part we introduced OpenStack AZs and presented different OpenShift deployment options regarding AZs.

In this post we will explain the ‘best case scenario’, using OpenShift on OpenStack with multiple Nova AZs and multiple Cinder AZs where the AZ names match.

PART II - Scenario One (Recommended): Multi Nova and Cinder AZs

The following scenario consists of:

  • 3 Nova AZs (AZ1, AZ2, AZ3)

  • 3 Cinder AZs (AZ1, AZ2, AZ3)

For demonstration, we will use the asb-etcd pod as an example as it is created at installation time. It is a pod that requires a volume to store data, thus creating a PVC. The purpose of this scenario is to show the asb-etcd pod is created in a Nova AZ that has the same name as the Cinder AZ used to create PVCs. If there are no nodes available within the AZ to mount the volume, the pod cannot start.

We begin by investigating the StorageClass created at installation time. It includes information about the provisioner (Cinder) and some other parameters. It looks like this:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  ...
parameters:
  fstype: xfs
provisioner: kubernetes.io/cinder

After a successful installation, the pod has been created in the AZ2 and it is running in one of the AZ2 nodes (cicd-node-1):

asb-etcd-1-t8chg   1/1   Running   0       8m     10.130.2.3   cicd-node-1.cicd.com

If I delete the pod, it is rescheduled again to that same node (notice the different pod name):

asb-etcd-1-7ljpf   1/1   	Running   0      	19s   	10.130.2.4   cicd-node-1.cicd.com

If I drain and cordon that node (oc adm drain cicd-node-1.cicd.com), eventually it will be scheduled to another node available in AZ2 (in this case, cicd-infra-1.cicd.com):

asb-etcd-1-lp457   1/1   	Running   0      	2m    	10.128.4.6   cicd-infra-1.cicd.com
 

If I drain and cordon all the remaining nodes in AZ2 (so 0 nodes in AZ2 scheduleables), the pod won’t be able to be scheduled:

4s      5s       2      asb-etcd-1-s4cqc Pod                                                Warning   FailedScheduling     default-scheduler             0/4 nodes are available: 4 NoVolumeZoneConflict.

With no nodes available within the AZ2, this leaves the pod in a pending state, as expected, because the asb-etcd pod was only allowed to be scheduled within AZ2.

asb-etcd-1-s4cqc   0/1  Pending   0     41s

Scenario One Conclusion

Pods are scheduled to nodes where the AZ name for nova is the same as the cinder AZ name for the PV they use. If there is no available node in the AZ to mount the volume, the pod won’t be able to start.

But wait… What about when you have multiple Nova availability zones and just one Cinder availability zone? How do I handle the following scenario? The next blog in the series,  "multiple Nova AZs with a single Cinder AZ" shifts the focus to answer these questions. It will explain why scenario one (multiple Nova AZs and multiple Cinder AZs) is recommended, but also how to handle environments where multiple Cinder AZs are not available.


About the author

Browse by channel

automation icon

Automation

The latest on IT automation for tech, teams, and environments

AI icon

Artificial intelligence

Updates on the platforms that free customers to run AI workloads anywhere

open hybrid cloud icon

Open hybrid cloud

Explore how we build a more flexible future with hybrid cloud

security icon

Security

The latest on how we reduce risks across environments and technologies

edge icon

Edge computing

Updates on the platforms that simplify operations at the edge

Infrastructure icon

Infrastructure

The latest on the world’s leading enterprise Linux platform

application development icon

Applications

Inside our solutions to the toughest application challenges

Original series icon

Original shows

Entertaining stories from the makers and leaders in enterprise tech