Empowering federated learning with multicluster management

The modern era of AI training, particularly for large models, faces simultaneous demands for computational scale and strict data privacy. Traditional machine learning (ML) requires centralizing the training data, resulting in significant hurdles and effort concerning data privacy, security, and data efficiency/volume.

This challenge is magnified across heterogeneous global infrastructure in multicloud, hybrid cloud, and edge environments, so organizations must train models using the existing distributed datasets while protecting data privacy.

Federated learning (FL) addresses this challenge by moving the model training to the data. Remote clusters or devices (collaborators/clients) train models locally using their private data and only share model updates (not the raw data) back to a central server (aggregator). This helps protect data privacy from end-to-end. This approach is crucial for privacy sensitive or high data load scenarios which we find in healthcare, retail, industrial automation, and software defined vehicles (SDV) with advanced driver-assistance systems (ADAS) and autonomous driving (AD) functionality, such as lane departure warning, adaptive cruise control, and driver fatigue monitoring.

To manage and orchestrate these distributed computation units, we utilize the federated learning custom resource definition (CRD) of Open Cluster Management (OCM).

OCM: The foundation for distributed operations

OCM is a Kubernetes multicluster orchestration platform and an open source CNCF Sandbox project. 

OCM employs a hub-spoke architecture and uses a pull-based model.

1. Hub cluster: This acts as the central control plane (OCM Control Plane) responsible for orchestration.
2. Managed (spoke) clusters: These are remote clusters where workloads are deployed.

Managed clusters pull their desired state from and report status back to the hub. OCM provides APIs like ManifestWork and Placement to schedule workloads. We’ll cover more federated learning API details below.

We'll now look at why and how the distributed cluster management design of OCM aligns closely with the requirements of deploying and managing FL  contributors.

Native integration: OCM as the FL orchestrator

1. Architectural alignment

The combination of OCM and FL is effective due to their fundamental structural congruence. OCM natively supports FL because both systems share an identical foundational design: the hub-spoke architecture and a pull-based protocol.

2. Flexible placement for multiactor client selection

OCM’s core operational advantage is its ability to automate client selection in FL setups by leveraging its flexible cross-cluster scheduling capabilities. This capability uses the OCM Placement API to implement sophisticated, multicriteria policies, providing efficiency and privacy compliance simultaneously.

The Placement API enables integrated client selection based on the following factors:

• Data locality (privacy criterion): FL workloads are scheduled only to managed clusters that claim to have the necessary private data.
• Resource optimization (efficiency criterion): The OCM scheduling strategy offers flexible policies that enable the combined assessment of multiple factors. It selects clusters not only based on data presence but also on advertised attributes like CPU/memory availability. 

3. Secure communication between collaborator and aggregator via OCM add-on registration

The FL add-on collaborator is deployed on the managed clusters and leverages OCM’s add-on registration mechanism to establish protected, encrypted communication with the aggregator on the hub. Upon registration, each collaborator add-on automatically obtains certificates from the OCM hub. These certificates authenticate and encrypt all model updates exchanged during FL, enabling confidentiality, integrity, and privacy across multiple clusters.

This process efficiently assigns AI training tasks only to adequately resourced clusters, providing integrated client selection based on both data locality and resource capacity.

The FL training lifecycle: OCM-driven scheduling

A dedicated Federated Learning Controller was developed to manage the training lifecycle of FL across multiple clusters. The controller utilizes CRDs to define the workflows and supports popular FL runtimes like Flower and OpenFL, and is extensible.

The OCM-managed workflow proceeds through defined stages:

Red Hat Advanced Cluster Management : Enterprise control and operational value for FL environments

The core APIs and architecture provided by OCM serve as the foundation of Red Hat Advanced Cluster Management for Kubernetes. Red Hat Advanced Cluster Management provides lifecycle management for a homogeneous FL platform (Red Hat OpenShift) across a heterogeneous infrastructure footprint.  Running the FL controller on Red Hat Advanced Cluster Management provides additional benefits beyond what OCM alone offers. Red Hat Advanced Cluster Management delivers centralized visibility, policy-driven governance, and lifecycle management across multicluster estates, significantly enhancing the manageability of distributed and  FL environments.

1. Observability

Red Hat Advanced Cluster Management provides unified observability across distributed FL workflows, enabling operators to monitor training progress, cluster status, and cross-cluster coordination from a single, consistent interface.

2. Enhanced connectivity and security

The FL CRD supports protected communication between the aggregator and clients through TLS-enabled channels. It also offers flexible networking options beyond NodePort—including LoadBalancer, Route, and other ingress types—providing protected and adaptable connectivity across heterogeneous environments.

3. End-to-end ML lifecycle integration with Red Hat Advanced Cluster Management and Red Hat OpenShift AI

By leveraging Red Hat Advanced Cluster Management with OpenShift AI, enterprises can build a complete FL workflow—from model prototyping and distributed training to validation and production deployment—within a unified platform.

Wrap up

FL is transforming AI by moving model training directly to the data, effectively resolving the friction between computational scale, data transfer, and strict privacy requirements. Here we've highlighted how Red Hat Advanced Cluster Management provides the orchestration, protection, and observability needed to manage complex distributed Kubernetes environments.

Get in touch with Red Hat today to explore how you can empower your organization with federated learning.