Introducing Selectable Profiles for etcd

In a Kubernetes cluster, the control plane's stability, resiliency, and performance drive the overall cluster stability, resiliency, and scalability. Generally speaking, the various components of Kubernetes are stateless. For storing object or artifact definitions of a cluster, the etcd key-value store acts as the single source of truth and as the only stateful component. This distributed key-value store is both reliable and strongly consistent. The APIServer, part of the control plane components, is the sole Kubernetes component directly interacting with etcd. All other components use the APIServer to retrieve or set information on the etcd. For this reason, losing etcd would lead to a catastrophic failure of the cluster.

The OpenShift control plane is defined by three supervisor nodes hosting the Kubernetes control plane components, including the etcd. To balance Failure Tolerance and Write Performance, OpenShift uses pre-defined etcd profiles to set the ETCD_HEARTBEAT_INTERVAL and ETCD_ELECTION_TIMEOUT for etcd. These profiles are optimized for the specific type of underlying infrastructure (e.g., cloud provider, virtualization, baremetal, etc.). On infrastructures with high variability of network latency or storage latency, OpenShift automatically uses settings with higher timers.

Internally, the OpenShift etcd Operator has two validated profiles:

1. Default profile: This profile applies the default etcd timers configuration
   1. Heartbeat Interval=100ms
   2. Election Timeout=1000ms
2. Slow profile: This profile is optimized for infrastructure with high variability on network or storage latency
   1. Heartbeat Interval=500ms
   2. Election Timeout=2500ms

Until now, these profiles have been automatically set by the OpenShift etcd Operator.

In some cases, a cluster administrator may need to select a slower or faster profile. This could be necessary when working with servers that have traditional hard disk drives (HDDs), deploying on unstable or saturated networks, or deploying a cluster that spans multiple locations. While these situations may not be ideal for Kubernetes, certain country or industry regulations or limitations from legacy application dependencies may require OpenShift to operate under these conditions. With this in mind, OpenShift 4.14 introduces the selectable profiles for etcd with the option of “controlPlaneHardwareSpeed”. With this, a cluster administrator can change the etcd profile to better match their environment.

The valid values for the new “controlPlaneHardwareSpeed” attribute in the OpenShift etcd Operator CR are:

• "" (empty string): This is the default option. Maintains the traditional behavior of the openshift-etcd-operator which allows the system to decide which speed to use. Clusters upgraded to Openshift 4.14 will initially operate here.
• “Standard”: This profile applies the default etcd timers configuration:
  • Heartbeat Interval=100ms
  • Election Timeout=1000ms
• “Slower”: This applies an etcd timers configuration to operate in unfavorable storage and network conditions:
  • Heartbeat Interval=500ms
  • Election Timeout=2500ms

The cluster administrator should be cautious when selecting profiles for openshift-etcd-operator, as higher etcd timer values increase recovery time for control plane failures. Whenever possible, keep the default behavior.

# Change the value. Valid values are "", "Standard", "Slower"
# The default value is "" (empty string)
oc patch etcd/cluster --type=merge \
-p '{"spec": {"controlPlaneHardwareSpeed": "<VALUE>"}}'

# To set the Standard profile
oc patch etcd/cluster --type=merge \
-p '{"spec": {"controlPlaneHardwareSpeed": "Standard"}}'

# To set the Slower profile
oc patch etcd/cluster --type=merge \
-p '{"spec": {"controlPlaneHardwareSpeed": "Slower"}}'

# To return to default behavior
oc patch etcd/cluster --type=merge \
-p '{"spec": {"controlPlaneHardwareSpeed": ""}}'

The changes will be detected by the OpenShift etcd operator environment variable controller triggering a rolling out update of an updated etcd. Note: This action may cause a temporary service disruption during the rollout process for the new etcd instance.

The ability to choose the etcd timers profile to use has been a capability requested by customers deploying OpenShift on unfavorable storage or networking conditions. This is what the new “controlPlaneHardwareSpeed” configuration option for the OpenShift etcd operator brings for the cluster administrators. If the organization has a use case that can benefit from it, try it in your non-production environments first, and let us know how it works for you.