Skip to main content

How to enable cross-cluster networking in Kubernetes with the Submariner add-on

The Submariner add-on for Red Hat Advanced Cluster Management (ACM) allows direct connectivity between pods and services on different Kubernetes clusters.
Silver chain

Photo by Pixabay from Pexels

In my previous article, How to back up Kubernetes persistent volumes with the VolSync add-on, I described using VolSync to back up persistent volumes in Kubernetes. This article explains how to use Submariner, which is also a Red Hat Advanced Cluster Management (ACM) for Kubernetes add-on. As Submariner's website explains:

Submariner enables direct networking between Pods and Services in different Kubernetes clusters, either on-premises or in the cloud.

Why Submariner?

As Kubernetes gains adoption, teams are finding they must deploy and manage multiple clusters to facilitate features like geo-redundancy, scale, and fault isolation for their applications. With Submariner, your applications and services can span multiple cloud providers, datacenters, and regions.

Submariner is completely open source and designed to be network plugin (CNI) agnostic.

What Submariner provides

This article serves a few purposes. I'll show the Submariner installation process first and then change the VolSync configuration I covered in my previous article. Specifically, I'm changing from VolSync using a load balancer such as MetalLB to Submariner to allow easier networking between pods.

Assume you have a limited number of external IP addresses you can assign. With the MetalLB solution, each ReplicationDestination would need an ExternalIP resource. If another team manages the IP space in your organization, it might take some time to get these IPs. If your cluster is directly on the internet (not typical), you may need to purchase additional IPs.

Submariner eliminates this IP problem and solves several other issues. The configuration steps in this article are based on section of the documentation on the Red Hat ACM site.

[ Learning path: Getting started with Red Hat OpenShift Service on AWS (ROSA)

You must have the following prerequisites before using Submariner:

  • You need a credential to access the hub cluster with cluster-admin permissions.
  • IP connectivity must be configured between the Gateway nodes. When connecting two clusters, at least one of the clusters must be accessible to the Gateway node using its public or private IP address designated to the Gateway node. Access Submariner NAT traversal for more information.
  • If you are using OVN-Kubernetes, clusters must be at Red Hat OpenShift Container Platform (OCP) version 4.11 or later.
  • If your OCP clusters use OpenShift SDN CNI, the firewall configuration across all nodes in each of the managed clusters must allow 4800/UDP in both directions.
  • The firewall configuration must allow 4500/UDP and 4490/UDP on the gateway nodes for establishing tunnels between the managed clusters.

    Note: This is configured automatically when your clusters are deployed in an Amazon Web Services, Google Cloud Platform, Microsoft Azure, or Red Hat OpenStack environment, but must be configured manually for clusters on other environments and for the firewalls that protect private clouds. (Learn the differences between managed services, hosted services, and cloud services.)

  • The managedcluster name must follow the DNS label standard defined in RFC 1123 and meet the following requirements:

    • Contain 63 characters or fewer
    • Contain only lowercase alphanumeric characters or -
    • Start with an alphanumeric character
    • End with an alphanumeric character

Since I'm currently running all single-node (SNO) (bare-metal) clusters, the gateway node is each SNO host. A site-to-site VPN allows direct connectivity between each of the clusters. These clusters are based on OCP 4.11 using OVNKubernetes.

You must open some additional, required ports if there is a firewall between the clusters:

  • IPsec NATT:
    • Default value: 4500/UDP
    • Customizable
  • VXLAN:
    • Default value: 4800/UDP
    • Not customizable
  • NAT discovery port:
    • Default value: 4490/UDP
    • Not customizable

I will cover the following topics:

  1. Installing the Submariner add-on to spoke1/spoke2 clusters
  2. Enabling Service Discovery
  3. Changing VolSync to use ClusterIP instead of LoadBalancer

[ Learn how to build a flexible foundation for your organization. Download An architect's guide to multicloud infrastructure. ]

1. Install the Submariner add-on to the spoke1 and spoke2 clusters

I will deploy Globalnet, a component that supports overlapping CIDRs in connected clusters. It is required because I used the default settings for my Services/ClusterIPs on each cluster. The default setting is I use a virtual network and NAT (provided by Globalnet) to allow the services on each cluster to talk.

You can find the YAML code for this article in my GitHub repo.

  1. Create the ManagedClusterSet resource called submariner. spoke1 and spoke2 will become members of it. Run the resource from the hub cluster.
    kind: ManagedClusterSet
      name: submariner
    oc apply -f mcs-submariner.yaml
  2. Now, create the broker configuration on the hub.
    kind: Broker
         name: submariner-broker
         namespace: submariner-broker
         globalnetEnabled: true
    oc apply -f submariner-broker.yaml
  3. Label spoke1 and spoke2 as part of the submariner ManagedClusterSet created in step 1.
    oc label managedclusters spoke1 "" --overwrite
    oc label managedclusters spoke2 "" --overwrite
  4. Create the submariner configuration for spoke1 and spoke2. This also runs from the hub. The VXLAN driver works best in my environment; I assume this is due to an issue with double encryption/encapsulation with my site-to-site VPN.
    kind: SubmarinerConfig
        name: submariner
        namespace: spoke1
      cableDriver: vxlan
    oc apply -f submariner-config-spoke1.yaml
    kind: SubmarinerConfig
        name: submariner
        namespace: spoke2
      cableDriver: vxlan
    oc apply -f submariner-config-spoke2.yaml
  5. Create the submariner ManagedClusterAddon for spoke1 and spoke2 on the hub.
    kind: ManagedClusterAddOn
         name: submariner
         namespace: spoke1
         installNamespace: submariner-operator
    oc apply -f mca-spoke1.yaml
    kind: ManagedClusterAddOn
         name: submariner
         namespace: spoke2
         installNamespace: submariner-operator
    oc apply -f mca-spoke2.yaml
  6. After you run these two commands, check the status.
    oc get ManagedClusterAddons -A|grep submariner

To check that the ManagedClusterAddon was enabled, make sure the output shows as True for submariner.

oc get command results
(Keith Calligan, CC BY-SA 4.0)

Next, check if the appropriate workloads were enabled on spoke1 and spoke2 to support Submariner.

oc -n spoke1 get managedclusteraddons submariner -oyaml

Wait a few minutes and check the conditions.

(Keith Calligan, CC BY-SA 4.0)
  • Ensure that the SubMarinerAgentDegraded shows False.
  • Ensure that the SubMarinerConnectionDegraded shows False.

[ Become a Red Hat Certified Architect and boost your career. ]

Another way to check its status is by using the subctl utility. You may need to do a few things to download this tool and run it from your bastion/jump-host. Copy the following to /usr/local/bin:

# Install podman if not already installed
yum install podman -y
# Login to Redhat registry
podman login
# Download image and extract tar.xz file to /tmp
oc image extract --path="/dist/subctl-v0.14*-linux-amd64.tar.xz":/tmp/ --confirm
# Untar/gz the file 
tar -C /tmp/ -xf /tmp/subctl-v0.14*-linux-amd64.tar.xz
# Copy subctl binary to /usr/local/bin and change permissions
install -m744 /tmp/subctl-v0.14*/subctl-v0.14*-linux-amd64 /usr/local/bin/subctl

From one of the clusters in the ManagedClusterSet (I picked spoke2), run the following command:

subctl show connections
Show connections
(Keith Calligan, CC BY-SA 4.0)

This command shows that sno-spoke1 is connected to spoke2.

[ Learn about upcoming webinars, in-person events, and more opportunities to increase your knowledge at Red Hat events. ]

2. Enable ServiceDiscovery

The documentation provides the following example to test whether ServiceDiscovery is working.

1. Begin with spoke2. The following commands download an unprivileged Nginx workload to the default project/namespace and expose it as a service on port 8080:

oc -n default create deployment nginx --image=nginxinc/nginx-unprivileged:stable-alpine

oc -n default expose deployment nginx --port=8080

Enable ServiceDiscovery on Nginx pod:

subctl export service --namespace default nginx

2. Change to spoke1 and attempt to reach this service. Create a nettest project and then a deployment.

oc new project nettest
kind: Deployment
apiVersion: apps/v1
  name: nettest
  replicas: 1
      app: nettest
      creationTimestamp: null
        app: nettest
        deploymentconfig: nettest
        - name: nettest
            - sleep
            - infinity
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          imagePullPolicy: Always
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
      imagePullSecrets: []
    type: RollingUpdate
      maxSurge: 25%
      maxUnavailable: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
  paused: false
oc create -f deployment.yaml

Find the name of the pod and remote shell (RSH) into it.

oc get po command
(Keith Calligan, CC BY-SA 4.0)
curl the site
(Keith Calligan, CC BY-SA 4.0)

This output proves that network connectivity and DNS resolutions work across this ClusterSet.

3. Change VolSync to use ClusterIP instead of LoadBalancer

In my VolSync article, I used a load balancer (MetalLB) resource to expose the replicationDestinationResource. I will change this to use ClusterIP and expose the service across the ClusterSets.

1. Review the replicationDestination service on spoke2. In my cluster, the project is called myopenshiftblog.

oc project myopenshiftblog
oc get svc
oc get svc
(Keith Calligan, CC BY-SA 4.0)

Since this is type LoadBalancer, the replicationDestinationResource must be changed to use serviceType: ClusterIP.

# Get appropriate replicationDestination if there is more than one
oc get replicationDestination
oc edit replicationDestination myopenshiftblog
edit the service type
(Keith Calligan, CC BY-SA 4.0)

2. Run the following command. Specify the namespace and servicename of the volsync service.

# Get the svc name
oc get svc
subctl export service --namespace myopenshiftblog volsync-rsync-dst-myopenshiftblog

There should be another svc exposed in addition to the regular svc.

oc get svc
oc get svc
(Keith Calligan, CC BY-SA 4.0)

A GlobalNetIP was assigned.

3. Test connectivity from the spoke1 cluster to this resource.

Assuming you still have the nettest pod deployed, run the following from the RSH prompt:

dig volsync-rsync-dst-myopenshiftblog.myopenshiftblog.svc.clusterset.local
dig for the site
(Keith Calligan, CC BY-SA 4.0)

The address now resolves.

4. On spoke1, change the replicationSource to the new DNS-based ClusterSet resource (volsync-rsync-dst-myopenshiftblog.myopenshiftblog.svc.clusterset.local). Edit the rsync.Address spec:

oc project myopenshiftblog
oc edit replicationsource myopenshiftblog
volsync information
(Keith Calligan, CC BY-SA 4.0)

5. Since I set this cronjob to every 3 minutes, it should run:

oc get replicationsource myopenshiftblog
oc get replication source
(Keith Calligan, CC BY-SA 4.0)

This example shows a successful sync based on the new DNS name (based on submariner/Globalnet).

[ Kubernetes: Everything you need to know ]

Wrap up

Submariner provides an alternative to the VolSync configuration I covered in my previous article. It enables pod and service networking, offering an additional option to solutions such as the MetalLB load balancer.

This originally appeared on myopenshiftblog and is republished with permission.

Topics:   Kubernetes   OpenShift   Resource management  
Author’s photo

Keith Calligan

Keith Calligan is an Associate Principal Solutions Architect on the Tiger Team specializing in the Red Hat OpenShift Telco, Media, and Entertainment space. He has approximately 25 years of professional experience in many aspects of IT with an emphasis on operations-type duties. More about me

Try Red Hat OpenShift, the enterprise Kubernetes application platform for big ideas.


Privacy Statement