Two-node OpenShift with fencing improves reliability at the edge

Edge computing environments present distinct hurdles as companies move processing capabilities nearer to where data is generated. Customers across industries, especially in retail, industrial, and telecommunications sectors, are increasingly demanding high availability for their edge deployments.

However, achieving high availability traditionally requires a three-node cluster to establish a reliable quorum. The primary factor driving organizations toward alternative topologies for large-scale edge deployments is the prohibitive cost of powering, maintaining, and deploying a third node across hundreds or thousands of sites—a motivation that has only grown stronger in light of the recent steep increase in hardware prices. 

To address this demand, we are introducing Red Hat OpenShift topologies targeting two nodes for edge deployments. While two-node OpenShift with arbiter offers a path that uses a small arbiter device to maintain quorum, it technically remains a three-node architecture. For cost-sensitive customers who require a strict, localized two-node footprint, Red Hat now offers two-node OpenShift with fencing. 

With the release of version 4.22 of OpenShift, two-node OpenShift with fencing is now generally available.

Two-node OpenShift with fencing architecture

Two-node OpenShift with fencing is a true two-node solution designed from the ground up for edge environments. Currently supported exclusively on x86 bare-metal platforms, two-node OpenShift with fencing also fully supports Red Hat OpenShift Virtualization, allowing teams to run both containerized and virtualized workloads on a minimal footprint.

Figure 1 provides an architectural overview of a two-node OpenShift with fencing deployment:

Figure 1. Architecture overview diagram of two-node OpenShift with fencing

Unlike a traditional setup that relies on a third node for tie-breaking, two-node OpenShift with fencing establishes high availability for the etcd database by relying on proven technologies in the Red Hat Enterprise Linux High Availability Add-On. Specifically, it integrates Corosync and Pacemaker into the OpenShift architecture.

This architecture offers flexible and resilient storage options, supporting node-local solutions like logical volume manager storage. Red Hat has also partnered with leading vendors to provide shared and distributed software-defined storage (SDS) options for two-node OpenShift with fencing. These include LINBIT (based on DRBD), Arctera InfoScale, and IBM Storage Fusion Data Foundation (based on Ceph). To provide data safety, I/O operations rely heavily on the fencing architecture; storage operations will completely halt upon a node failure and will only resume after a successful fencing operation has completed.

The concept of fencing: Preventing split-brain scenarios

In any even-numbered cluster, the loss of communication between nodes introduces the risk of a split-brain scenario. If the network link between the two nodes is severed, each node might independently assume the other has failed and attempt to take control of the workloads. If both run the same workload or write to storage simultaneously, catastrophic data corruption can occur. Imagine suddenly facing 2 instances of the same database, both containing live customer transactions. Merging them back into a single consistent instance would be a nightmare. 

A concept known as fencing offers protection against split-brain situations. Fencing imposes consistency by requiring that Pacemaker on the surviving node forcefully powers down or reboots the failed or disconnected node.

To execute this, two-node OpenShift with fencing requires the underlying hardware to possess a baseboard management controller (BMC) that supports the RedFish API. When communication with a node is lost, Corosync declares a failure state. The node is only potentially failed, because the node's actual state is uncertain. It may still be running workloads and processing requests. 

To get into a defined state, Pacemaker initiates the fencing operation. The fencing operation sends a ForceRestart command to the BMC of the failed node. That performs an immediate, nongraceful shutdown followed by a system power-on. Once that command has executed successfully, the system is in a well-defined and consistent state: what had been a potentially failed workload is now guaranteed to be stopped. That includes the OpenShift internal etcd database.

Once a node successfully uses the BMC to fence its peer, it safely reconfigures etcd into a cluster of one, regains quorum, and restarts the workloads.

A question that may occur to you at this point: How can we prevent a race condition in which both nodes attempt to fence each other simultaneously? This is done heuristically by establishing different delay times between failure detection and the start of the fencing operation on each node. 

Walkthrough: A node fails and is fenced

To fully understand the power of two-node OpenShift with fencing, let's walk through an example description of a single node failing and being fenced, as depicted in Figure 2.

Figure 2. Sequence of events in a single-node failure

Phase 1: Steady state

Under normal operating conditions, both Node 1 and Node 2 are online and functioning. The etcd database is operating as a cluster of 2, and Corosync maintains a stable view of the system. The OpenShift control plane is responsive, quorum is securely established, the scheduler is operational, and workloads are running healthily.

Phase 2: The failure event 

Node 1 suffers an unexpected hardware failure and goes offline. Within seconds, etcd loses its quorum and stops responding. Consequently, the API server and the control plane become unavailable. To avoid unnecessary fencing operations during brief network hiccups, the cluster does not react immediately or impulsively. Instead, Corosync simply begins to log missed heartbeats from Node 1. The cluster's workload status shifts to "degraded." That is because some workload that has been running on Node 1 is now missing, not running. In some scenarios, that might cause service degradation—if it's a single instance database or virtual machine, for instance. 

Phase 3: Fencing initiated  

If the outage persists over several minutes, Corosync officially declares a failure state. Pacemaker takes over and initiates the fencing operation. Using the hardware's BMC, Pacemaker issues a RedFish API call to aggressively reboot Node 1. During this active fencing phase, the control plane, scheduler, console, etc., remain temporarily unavailable. But the workload pods on the remaining node continue to run and serve requests. 

Phase 4: Recovery and regaining quorum

Recovery will only proceed once the fencing operation has been executed successfully—that is, when it has returned a 200 OK status code. With Node 1 confirmed fenced, the surviving Node 2 reconfigures etcd into a cluster of one. By doing so, Node 2  regains quorum. The control plane assumes a degraded but functional state, and the scheduler becomes operational once more. Assuming there is available capacity, the scheduler immediately restarts the missing workloads on Node 2.

Phase 5: Resuming normal operations 

Eventually, once the hardware issue is resolved, Node 1 comes back online. The cluster detects the node and automatically reconfigures etcd back into a cluster of two via a learner phase. Node 1 receives all the etcd transactions missed during the outage period. This can be pretty I/O intensive, which is why two-node OpenShift with fencing requires disks with SSD-quality performance or better. 

The cluster then resumes full normal operations, re-establishing high availability with a responsive control plane and a healthy workload status.

Resilience in extreme conditions

Two-node OpenShift with fencing is built to handle highly complex edge failure modes beyond simple network drops. However, some extreme conditions might require manual intervention:

• Total outage and cold boot: In the event of a total site power failure where both nodes go down, a single node returning online will not start workloads. Because it lacks quorum, it cannot know if the other node possesses a more recent state. There are 2 options to resolve this:
  1. The communication with the originally failed node is re-established. Then the two nodes can determine which has the most recent state and should become the leader. The other node then will rejoin the cluster via a learner phase recovery phase, as described above.
  2. In the second option, an admin must intervene manually via SSH to confirm the state of the other node, and make sure it has been offline for the whole outage period. That’s why it is important for two-node OpenShift with fencing deployments to include properly configured SSH access—by providing public keys during installation, for instance. Please note that in this state, the oc debug node/nodeXXX command will not work, as the control plane is still offline.
• Reboot of a survivor (double error): If the cluster is already degraded (running on only one node after a successful fence) and the surviving node reboots (e.g., due to a power outage), you cannot safely guarantee consistency upon restart. The other node might have been active during the outage. The state is unknown.

As in a cold boot scenario, this requires the second node to start up to re-establish communication, or manual administrative intervention via SSH.

Summary

Two-node OpenShift with fencing provides highly available OpenShift on only two nodes while protecting consistency. If you want to learn more, you can explore an interactive demo, check out the product documentation, or talk to a Red Hatter.