Subscribe to the feed

Open Cluster Management

Open Cluster Management is a community-driven project that is focused on multicluster and multicloud scenarios for Kubernetes applications. Open APIs are included and evolving within this project for cluster registration, work distribution, dynamic placement of policies and workloads, and more.

The Open Cluster Management add-on enables any capability within the Kubernetes ecosystem to orchestrate itself across multiple clusters and cloud providers. Open Cluster Management provides core primitives to satisfy the previously mentioned requirements to ease the multicluster enablement.

In this blog, you will learn how to develop a custom add-on, which enables you to extend your custom actions by performing them on multiple clusters.

Hub cluster

The hub cluster is the common term that is used to define the central controller that runs on a Kubernetes cluster. The hub cluster aggregates information from multiple clusters by using an asynchronous work request model.

Managed cluster

The managed cluster provides a representation of the managed cluster on the hub cluster. The ManagedCluster resource controls whether or not the remote cluster is accepted by the hub cluster for management. After the managed cluster is accepted, it can retrieve information from the hub cluster to direct a set of manifests or actions to apply.

What is an add-on?

Open Cluster Management has a way to help you create an extension based on the foundation components so you can work with multiple clusters in different ways. Some examples of add-ons include:

  • A tool to collect alert events in the managed cluster and send them to the hub cluster.

  • A network solution that uses a hub to share the network information and establish a connection across managed clusters.

  • A tool to run security policies on multiple clusters.

In general, if a management tool needs a different configuration for each managed cluster or a secured communication between managed cluster and hub cluster, it can use the add-on feature of the Open Cluster Management to simplify the installation and improve day two operations.

Normally, an add-on contains two components:

  1. The addon-hub-controller, which is a controller that runs on the hub cluster.
  2. The addon-agent, which is an agent that runs on the managed cluster.

The following diagram shows how the two components fit in to the architecture of the add-on.


As shown in the previous diagram, some foundation components from Open Cluster Management are also required to manage the lifecycle of an add-on. The following list describes those components:

  • registration-hub-controller: This controller on the hub cluster updates the add-on status according to the status of the cluster's lease on the hub.
  • registration-agent: This agent on the managed cluster helps to register the add-on with the hub cluster and creates a secret that contains the hub kubeconfig for the add-on on the managed cluster. The agent also continues updating the add-on status on the hub cluster based on the add-on lease created on the managed cluster.
  • work-agent: This agent on the managed cluster distributes a list of manifests from the hub cluster to the managed cluster and applies the manifests on the managed cluster for the add-on.

How to develop an add-on

The easiest way to build an add-on is to leverage addon-framework, which is a library that contains the necessary interfaces and default implementations for add-on lifecycle management. In this section, you can develop an example helloworld add-on.

Step 1: Build your add-on hub controller

  1. Start with the AgentAddon interface to develop an addon-hub-controller.

    // AgentAddon defines manifests of agent deployed on managed cluster
    type AgentAddon interface {
    // Manifests returns a list of manifest resources to be deployed on the managed cluster for this add-on
    Manifests(cluster *clusterv1.ManagedCluster, addon *addonapiv1alpha1.ManagedClusterAddOn) ([]runtime.Object, error)
    // GetAgentAddonOptions returns the agent options.
    GetAgentAddonOptions() AgentAddonOptions

The Manifests method is expected to return the manifest resources that are required to deploy the addon-agent on a managed cluster. The GetAgentAddonOptions method returns add-on configuration including add-on name and registration option.

  1. Create a struct named helloWorldAgent to implement the interface.

    type helloWorldAgent struct {
    kubeConfig *rest.Config
    recorder events.Recorder
    agentName string
    var _ agent.AgentAddon = &helloWorldAgent{}
  2. Create an AddonManager, which is provided by addon-framework, and register the agent that you built before starting it.

    mgr, err := addonmanager.New(controllerContext.KubeConfig)
    if err != nil {
    return err
    agentRegistration := &helloWorldAgent{
    kubeConfig: controllerContext.KubeConfig,
    recorder: controllerContext.EventRecorder,
    agentName: utilrand.String(5),

Step 2: Build your add-on agent

  1. Create an agentController. This controller monitors configmaps in the cluster namespace on the hub cluster, and copies them to the default namespace on the managed cluster.

    type agentController struct {
    spokeKubeClient kubernetes.Interface
    hunConfigMapLister corev1lister.ConfigMapLister
    clusterName string
    recorder events.Recorder
  2. Start an agentController and a LeaseUpdater. The LeaseUpdater creates a lease for the add-on in the add-on installation namespace and updates it every 60 seconds.

    // create an agent contoller
    agent := newAgentController(
    // create a lease updater
    leaseUpdater := lease.NewLeaseUpdater(
    go agent.Run(ctx, 1)
    go leaseUpdater.Start(ctx)

You just finished a major part of the helloworld add-on development. Visit the helloworld directory in GitHub to view the complete source code. Next, you can enable it.

How to enable your add-on

After you finish developing your add-on, you need to enable it on your managed cluster by using the ManagedClusterAddOn API. You can deploy the helloworld add-on using Open Cluster Management.

  1. Follow the getting started instructions of registration-operator to prepare a kind cluster and deploy the Open Cluster Management instance on the kind cluster to manage itself.

    After you deploy Open Cluster Management successfully, a managed cluster (cluster1) and its certificate signing request (CSR) are created on the hub:

    export KUBECONFIG=$HOME/.kube/config

    kubectl get managedcluster cluster1

    cluster1 true https://localhost True True 20m
  2. Accept the managed cluster and approve its CSR with the following command:

    kubectl patch managedcluster cluster1 -p='{"spec":{"hubAcceptsClient":true}}' --type=merge

    kubectl get csr -l | grep Pending | awk '{print $1}' | xargs kubectl certificate approve
  3. Build a docker image of the helloworld add-on on your local host:

    make images

    An image is created.

  4. Load the image to your kind cluster:

    kind load docker-image
  5. Deploy the helloworld add-on controller on the hub cluster:

    make deploy-example

    The pod of the helloworld add-on controller is created in the open-cluster-management namespace:

    kubectl -n open-cluster-management get pods

    helloworld-controller-5857d64c7c-qxnzs 1/1 Running 0 47m
  6. Enable the helloworld add-on on the managed cluster by using the ManagedClusterAddOn API:

    cat << EOF | kubectl apply -f -
    kind: ManagedClusterAddOn
    name: helloworld
    namespace: cluster1
    installNamespace: default

    The helloworld add-on is then installed in the default installation namespace on the managed cluster:

    kubectl -n default get pods
    helloworld-agent-79cf779b98-vgwz9 1/1 Running 0 36m

You enabled the helloworld add-on on the managed cluster. Next, try to use it.

How to use your add-on

After enabling your add-on, complete the following steps to use it:

  1. Create a configmap for helloworld in the managed cluster namespace cluster1 on the hub cluster:

    cat << EOF | kubectl apply -f -
    apiVersion: v1
    kind: ConfigMap
    name: helloworld
    namespace: cluster1
    value: helloworld
  2. The add-on synchronizes with the configmap of the managed cluster, you can find it in the YAML file of the managed cluster:

    kubectl -n default get cm helloworld -oyaml
    apiVersion: v1
    value: helloworld
    kind: ConfigMap
    creationTimestamp: "2021-07-06T06:42:04Z"
    - apiVersion: v1
    fieldsType: FieldsV1
    .: {}
    f:value: {}
    manager: helloworld
    operation: Update
    time: "2021-07-06T06:42:04Z"
    name: helloworld
    namespace: default
    resourceVersion: "32967"
    uid: f46d2e50-a29d-4b9b-9113-24219821b365

How does your helloworld add-on work?

Now that your add-on is enabled and working, you might be interested in the process that makes this add-on work. The order of the steps in the process doesn't matter, and some of the actions might happen simultaneously. After you applied the ManagedClusterAddOn called helloworld on the hub cluster, the following process starts:


  • On the hub cluster, the AddonManager that is started in the addon-hub-controller calls back to the Manifests method of the helloWorldAgent. This call back collects the manifest resources of the helloworld add-on and creates the manifest works on the managed cluster namespace.

    kubectl -n cluster1 get manifestworks
    addon-helloworld-deploy 8m27s
  • On the hub cluster, the AddonManager calls back to the GetAgentAddonOptions method of helloWorldAgent to get the registration information of the helloworld add-on and append it to the add-on status.

    kubectl -n cluster1 get managedclusteraddons helloworld -oyaml
    kind: ManagedClusterAddOn
    - signerName:
    - system:open-cluster-management:cluster:cluster1:addon:helloworld
    - system:open-cluster-management:addon:helloworld
    - system:authenticated
    user: system:open-cluster-management:cluster:cluster1:addon:helloworld:agent:8fmr7
  • On the managed cluster, the work-agent watches the manifestwork of the helloworld add-on and deploys the addon-agent of the helloworld add-on on the managed cluster.

    kubectl get clusterrolebindings helloworld-addon
    helloworld-addon ClusterRole/helloworld-addon 49m

    kubectl -n default get deploy,sa
    deployment.apps/helloworld-agent 1/1 1 1 37m

    serviceaccount/helloworld-agent-sa 1 37m
  • On the managed cluster, the registration-agent collects the registration information from the status of the helloworld add-on and creates a CSR on the hub cluster.

    kubectl get csr
    addon-cluster1-helloworld-pgmt4 8s system:open-cluster-management:cluster1:sdcnn Pending
  • On the hub cluster, the AddonManager calls back to the function CSRApproveCheck of the Registration configuration to determine whether the CSR should be approved.

    kubectl get csr
    addon-cluster1-helloworld-pgmt4 8s system:open-cluster-management:cluster1:sdcnn Approved,Issued
  • After the CSR is approved, the registration-agent on the managed cluster watches the approved CSR that is on the hub cluster and gathers the hub credentials from it. Then the registration-agent uses the hub cluster credentials to create a hub kubeconfig and saves it as a secret in the helloworld add-on installation namespace.

    kubectl -n default get secrets
    helloworld-hub-kubeconfig Opaque 3 5m45s
  • The addon-agent of the helloworld add-on mounts the secret and uses the hub cluster's kubeconfig of the secret to access to the hub kube-apiserver. In day two operation, when the certificate of the helloworld add-on is about to expire, the registration-agent sends a request to rotate the certificate on the hub cluster. The addon-hub-controller on the hub cluster automatically approves the certificate rotation request.

  • After the addon-agent of the helloworld add-on is deployed on the managed cluster, the agent creates a lease in its installation namespace.

    kubectl -n default get leases
    helloworld 7m52s
  • The registration-agent on the managed cluster uses this lease to update the status of the helloworld add-on on the hub cluster.

    kubectl -n cluster1 get managedclusteraddons helloworld -oyaml
    kind: ManagedClusterAddOn
    - lastTransitionTime: "2021-07-09T02:52:49Z"
    message: helloworld add-on is available.
    reason: ManagedClusterAddOnLeaseUpdated
    status: "True"
    type: Available

In this blog, we introduced an approach to extend the capability of Open Cluster Management by developing and using an add-on. By following these steps, you can develop a sample add-on based on addon-framework and deploy it on both the hub cluster and the managed cluster. It also explains how an add-on registers itself on the hub cluster and updates its status with the lease. With this knowledge, you are able to change some settings and build your own custom managed cluster add-ons.

About the authors


Browse by channel

automation icon


The latest on IT automation for tech, teams, and environments

AI icon

Artificial intelligence

Updates on the platforms that free customers to run AI workloads anywhere

open hybrid cloud icon

Open hybrid cloud

Explore how we build a more flexible future with hybrid cloud

security icon


The latest on how we reduce risks across environments and technologies

edge icon

Edge computing

Updates on the platforms that simplify operations at the edge

Infrastructure icon


The latest on the world’s leading enterprise Linux platform

application development icon


Inside our solutions to the toughest application challenges

Original series icon

Original shows

Entertaining stories from the makers and leaders in enterprise tech