Direct VM networking in AWS: Introducing BGP cloud connector on Red Hat OpenShift Service on AWS

When migrating virtual machines (VMs) to the cloud, virtualization engineers hit a familiar wall: how do you route traffic directly to a Red Hat OpenShift Virtualization guest VM using that VM’s own IP address?

OpenShift Virtualization has long offered a solution to this problem with localnets—as long as the cluster is using an on-premise Layer 2 network. Most cloud providers only expose Layer 3 on their networks, however, forcing OpenShift Virtualization customers in cloud environments to redesign their networking architecture. Without direct IP routing in the cloud, virtualization teams were forced to treat VMs like container workloads, adding NodePort Services and Routes that obscured the VMs’ IP addresses.

We are introducing a new solution: BGP cloud connector, launching for Red Hat OpenShift Service on AWS (ROSA) on cluster versions 4.22 or later. It allows customers to route traffic between an in-cluster overlay network (a ClusterUserDefinedNetwork, or CUDN) and the virtual private cloud (VPC) in which the cluster resides. Customers can then connect the VPC to their corporate networks using AWS’s existing solutions (like Transit Gateway or Direct Connect) to achieve end-to-end connectivity between any of their networks and guest VMs in a Red Hat OpenShift Service on AWS cluster.

Expanded workload possibilities

Because guest VMs can retain their native IP addresses externally, common challenges now have straightforward solutions. These include:

• Firewalls and access control that rely on a packet’s source IP address can now differentiate between each VM. This is because traffic from VMs is no longer passed through Network Address Translation (NAT) on worker nodes’ IP addresses.
• Active Directory Dynamic DNS can now correctly assign domain names to VM domain clients. This is because the Domain Controller sees the same IP address that clients report from their network interfaces.
• Clustered software, such as older versions of Valkey, that is designed around clients seeing the same IP address for server cluster members, can now work correctly. This is because machines inside and outside of your cluster all see the same IP address.

Demonstration

To see BGP cloud connector in action, let’s look at how an EC2 instance in a VPC communicates directly with a VM running inside Red Hat OpenShift Service on AWS without NAT.

Figure 1 illustrates a VM guest within my cluster. It has been assigned the IP address 192.168.5.3 from a CUDN that BGP cloud connector has configured the VPC to route to.

Figure 1: The VM guest with IP address 192.168.5.3

I’m running a chatbot on that VM, listening on port 8000.

[rhel@sample-vm ~]$ hostname -i
192.168.5.3
[rhel@sample-vm ~]$ socat TCP-LISTEN:8000,reuseaddr,fork EXEC:'/usr/bin/cla -i' # Expose the RHEL command-line assistant chatbot on port 8000

I also have an EC2 instance in the same VPC. I can hit my VM’s port 8000 directly on its own IP address from the EC2 instance to interact with that chatbot.

[ec2-user@ip-10-0-4-42 ~]$ socat - TCP:192.168.5.3:8000
Welcome to the interactive mode for command-line assistant! To exit, press Ctrl + C or type '.exit'.
The current session does not include running context.
>>> What is BGP? Be concise.
⁺₊+ Asking the command-line assistant...
────────────────────────────────────────────────────────────────────────
BGP (Border Gateway Protocol) is a standardized exterior gateway protocol designed to exchange routing and reachability information among autonomous systems (AS) on the internet. It is commonly used between Internet Service Providers (ISPs) to determine the best paths for traffic across the internet.
────────────────────────────────────────────────────────────────────────
Always review AI-generated content prior to use.

How BGP cloud connector works

The Red Hat OpenShift Service on AWS cluster’s worker nodes act as routers for traffic between the VPC and the CUDN that the guest VMs sit on.

The cluster uses Border Gateway Protocol (BGP) to advertise routes to Amazon VPC Route Server, which updates them in VPC route tables. BGP cloud connector keeps the set of Route Server peers up-to-date as nodes join and leave the cluster.

Each route configures the VPC to send traffic for the CUDN’s IP range to one of the cluster’s worker nodes. The worker node then forwards the traffic to the worker hosting the VM guest. Since VPC route tables only contain one route per IP prefix, Route Server chooses one route at a time to publish. This means that one worker node at a time acts as the router for all incoming traffic for a particular CUDN.

If the active gateway node drops offline, Route Server quickly detects the failure over BGP and updates the VPC route table to point to a healthy worker, keeping your VM traffic flowing.

Get started

If your architecture requires direct connectivity to VMs’ IP addresses, it’s straightforward to get started. Follow our product documentation for a detailed guide on how to set up your VPC, install and configure the operator, and deploy your first directly addressable VM.

If you don’t have a Red Hat OpenShift Service on AWS cluster yet, follow the  Quickstart Guide or the Learning Path on Getting Started.

If you’d like to learn more about OpenShift’s Virtualization’s networking concepts, check out the Red Hat webinar on the topic, “Networking OpenShift Virtualization: What you really need to know.”

Happy virtualizing!