Kubernetes Federation V2 on OpenShift 3.11

With datacenters spread across the globe, users are increasingly looking at ways to spread their applications and services across multiple locales or clusters. This need is driven by multiple use cases: from providing high availability, spreading load across multiple clusters while being resilient to individual cluster failures; to avoiding provider lock-in by using hybrid cloud solutions that have access to and make use of multiple clusters.

Red Hat has been working in the Kubernetes Multicluster Special Interest Group (SIG) and Federation Working Group to develop Kubernetes Federation V2 which is designed to allow users to deploy services and workloads to multiple clusters from a single API. Today, we're very excited to bring you a preview of Federation V2 on OpenShift 3.11.

Motivations

Our exploration of the multicluster problem space is motivated by the needs of our users whose use cases include::

• distribution of applications, services and policy to multiple clusters
• migration of applications and services and their storage between clusters
• disaster recovery for those applications and services

To address these needs and in order to be useful to the widest possible audience, we have designed with modularity in mind. This means we have added the ability to accept individual pieces and also alter the system behaviour. This will also be usable with custom resources.

Introducing Federation V2

Federation V2 is a Kubernetes operator leveraging Custom Resource Definitions that provides tools for managing applications and services in multiple Kubernetes clusters tracked by the Kubernetes Cluster Registry. Federation allows users to deploy workloads to clusters in the cluster registry, program DNS with information about those workloads, and dynamically adjust replicas in the different clusters a workload is deployed in. As Federation matures, we expect to add features dealing with storage, workload placement, etc.

Federation Concepts

Fundamentally, federation must be configured with two types of information:

• Which API types federation should handle
• Which clusters federation should target for distributing resources of those types

For each API type that federation handles, different parts of the declared state live in different API resources:

• A ‘template’ type holds the base specification of the resource.
• A ‘placement’ type holds the specification of the clusters the resource should be distributed to.
• An optional ‘overrides’ type holds the specification of how the template resource should be varied in some clusters.

Propagation refers to how resources are distributed to the target clusters. Currently, there is an implementation of an active reconciliation approach, where federation runs a controller that actively pushes resources to each target cluster.

Scheduling refers to a decision-making capability that can decide how workloads should be spread across different clusters similar to how a human operator would.

Finally, applications and services deployed in multiple clusters frequently require DNS records that route external requests to one of the servicing clusters. The DNS capability of federation maintains DNS entries for each endpoint for a federated service or ingress.

Example: Federating a Deployment across two clusters

For an example using the Deployment resource take a look at this video. The example describes a Deployment resource spread over 2 clusters, with 3 replicas in one cluster, and 5 replicas in the other.

Installation on OpenShift and example app

Instructions

Let’s take a look at an example using the Deployment resource. This example describes a Deployment resource spread over 2 clusters, with 3 replicas in one cluster, and 5 replicas in the other.

The basic definition of the Deployment lives in a FederatedDeployment:

apiVersion: core.federation.k8s.io/v1alpha1

kind: FederatedDeployment

metadata:

  name: test-deployment

  namespace: test-namespace

spec:

  template:

    metadata:

      labels:

        app: nginx

    spec:

      replicas: 3

      selector:

        matchLabels:

          app: nginx

      template:

        metadata:

          labels:

            app: nginx

        spec:

          containers:

          - image: nginx

            imagePullPolicy: Always

            name: nginx

A FederatedDeploymentPlacement resource of the same name contains information about the clusters the Deployment should exist in:

apiVersion: core.federation.k8s.io/v1alpha1

kind: FederatedDeploymentPlacement
 metadata:
 name: test-deployment
 namespace: test-namespace
 spec:
 clusternames:
 - cluster2
 - cluster1

A FederatedDeploymentOverrides resource of the same name contains information about how the replicas should be differentiated in certain clusters:

apiVersion: core.federation.k8s.io/v1alpha1

kind: FederatedDeploymentOverride
 metadata:
 name: test-deployment
 namespace: test-namespace
 spec:
 overrides:
 - clusterName: cluster2
 replicas: 5

At this time, it is only possible to override a single field for a given federated type (in the case of Deployments, the ‘replicas’ field).  If it is necessary that an override be applied on the initial creation of a target resource in member clusters, the Override resource should be created before the Template resource.

The Future

Our next steps in the Kubernetes community will be driven in part by feedback we receive on the developer preview for federation, and so we are extremely eager to hear any feedback our users may have. The best way to discuss feedback and get support is to engage with the upstream community:

• Get support from the community in #sig-multicluster on Kubernetes slack
• Open issues on the Federation V2 repo

In future blog posts, we’ll take detailed looks at the DNS, the scheduling features of Federation and storage.