Red Hat Advanced Cluster Management and ClusterImageSets

6 juillet 2022Jose Gato, Dangpeng Liu7 minutes (temps de lecture)

Red Hat Advanced Cluster Management (RHACM) is a solution that allows you to manage Kubernetes/OpenShift clusters and infrastructures. It includes OpenShift Hive, an operator that runs as a service based on OpenShift. The Hive service can be used to provision and perform initial configuration of OpenShift clusters. Hive enables deployments with different cloud providers, such as AWS, Azure, or Google Cloud Platform, as well as baremetal infrastructure.

OpenShift Hive provides a custom resource called ClusterImageSet, which points to different available OpenShift versions. You can easily check which available OpenShift versions are available with the following command:

> oc get clusterimagesets.hive.openshift.io 
NAME                      RELEASE
img4.10.3-x86-64-appsub   quay.io/openshift-release-dev/ocp-release:4.10.3-x86_64
img4.10.4-x86-64-appsub   quay.io/openshift-release-dev/ocp-release:4.10.4-x86_64
img4.10.5-x86-64-appsub   quay.io/openshift-release-dev/ocp-release:4.10.5-x86_64
img4.6.1-x86-64-appsub    quay.io/openshift-release-dev/ocp-release:4.6.1-x86_64
img4.6.12-x86-64-appsub   quay.io/openshift-release-dev/ocp-release:4.6.12-x86_64
img4.6.13-x86-64-appsub   quay.io/openshift-release-dev/ocp-release:4.6.13-x86_64
img4.6.15-x86-64-appsub   quay.io/openshift-release-dev/ocp-release:4.6.15-x86_64
img4.6.16-x86-64-appsub   quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64
img4.6.17-x86-64-appsub   quay.io/openshift-release-dev/ocp-release:4.6.17-x86_64
img4.6.18-x86-64-appsub   quay.io/openshift-release-dev/ocp-release:4.6.18-x86_64
img4.6.19-x86-64-appsub   quay.io/openshift-release-dev/ocp-release:4.6.19-x86_64
.....

The oc CLI provides the full list of available images. However, when using RHACM console, this list is automatically filtered to show only the latest versions. This encourages the creation of clusters with the latest release images but does not prevent use of previous versions. See the following command:

> oc get clusterimagesets.hive.openshift.io img4.10.10-x86-64-appsub -o jsonpath={.metadata.labels.visible}
false
> oc get clusterimagesets.hive.openshift.io img4.10.15-x86-64-appsub -o jsonpath={.metadata.labels.visible}
true

Subscribing ClusterImageSet

The list of available images is automatically synced using a combination of subscription and channel resources. See the following information about subscription manifests (simplified for better reading):

> oc get subscription hive-clusterimagesets-subscription-fast-0 -n open-cluster-management -oyaml
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
    annotations:
        apps.open-cluster-management.io/git-branch: release-2.5
        apps.open-cluster-management.io/git-path: clusterImageSets/fast
    creationTimestamp: "2022-03-29T12:39:46Z"
    labels:
        subscription-pause: "false"
    name: hive-clusterimagesets-subscription-fast-0
    namespace: open-cluster-management
spec:
    channel: open-cluster-management/acm-hive-openshift-releases-chn-0
    placement:
        local: true
status:
    lastUpdateTime: "2022-05-24T22:20:38Z"
    message: Active
    phase: Subscribed
    statuses:
    /:
        packages:
        acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.1-x86-64-appsub:
            lastUpdateTime: "2022-04-17T14:43:26Z"
            phase: Subscribed
        acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.3-x86-64-appsub:
            lastUpdateTime: "2022-04-17T14:43:25Z"
            phase: Subscribed
        acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.4-x86-64-appsub:
            lastUpdateTime: "2022-04-17T14:43:27Z"
            phase: Subscribed
        acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.6-x86-64-appsub:

See the following important information we can check for in the subscription:

The subscription uses a fast channel and the release 2.5 branch, which is the current version of RHACM in the example.
The subscription is active (not paused).
The lastUpdateTime shows the last update time for the subscription.
There is a list of all the ClusterImageSet received.

The spec.channel points to the corresponding channel resource, which keeps the communication with the repository containing all the ClusterImageSets. In this case, the subscription is using the open-cluster-management/acm-hive-openshift-releases-chn-0 RHACM channel.

See the following to get more info about the RHACM channel:

> oc get channel acm-hive-openshift-releases-chn-0 -n open-cluster-management -oyaml
apiVersion: apps.open-cluster-management.io/v1
kind: Channel
metadata:
    annotations:
        apps.open-cluster-management.io/reconcile-rate: low
    creationTimestamp: "2022-03-29T12:39:46Z"
    name: acm-hive-openshift-releases-chn-0
    namespace: open-cluster-management
spec:
    pathname: https://github.com/stolostron/acm-hive-openshift-releases.git
    type: Git

The channel points to a Git repository. This repository is maintained and updated by Red Hat with the different available OCP versions in the way of ClusterImageSets. See that the subscription uses this channel, selecting the git-repo and git-branch:

    apiVersion: apps.open-cluster-management.io/v1
    kind: Subscription
    metadata:
      annotations:
        apps.open-cluster-management.io/git-branch: release-2.5
        apps.open-cluster-management.io/git-path: clusterImageSets/fast

With this mechanism, RHACM is updated with all the available versions of RHOCP.

Combining the Git URL configured by the channel, the git-branch and the git-path from the subscription, you can browse to the repo, where the different ClusterImageSets are downloaded.

Changing the Channel to Sync

RHACM comes with the fast channel installed by default. This can be changed by following next steps. If you are just using RHACM, by default, all the resources should be managed into the open-cluster-management Namespace. See the following procedure:

Pause the fast channel subscription. Run the following command:

> oc -n open-cluster-management patch appsub hive-clusterimagesets-subscription-fast-0 --type='json' -p='[{"op":"replace","path": "/metadata/labels/subscription-pause","value":"true"}]'
subscription.apps.open-cluster-management.io/hive-clusterimagesets-subscription-fast-0 patched

Delete all the current available ClusterImageSets. These objects only contain references to download images, so you are not really deleting anything critical. The resources we are deleting belong to the fast channel. Ensure you did not manually create ClusterImageSets; these will be also deleted and you may want to keep them:

> oc delete clusterimageset --all
clusterimageset.hive.openshift.io "img4.10.10-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.11-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.12-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.13-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.14-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.15-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.3-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.4-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.5-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.6-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.8-x86-64-appsub" deleted

Add a new subscription for stable channel.

Best practice: Get the Manifest from your current subscription. Just clean the usual extra data created by OpenShift/Kubernetes.

Finally, change the git-patch, ensure it is not paused, and change the name for stable-0. See the following example based on the preinstalled fast subscription:

apiVersion: apps.open-cluster-management.io/v1                                    
kind: Subscription                                                                
metadata:                                                                      
    annotations:                                                                 
        apps.open-cluster-management.io/git-branch: release-2.5                    
        apps.open-cluster-management.io/git-path: clusterImageSets/stable          
    labels:         
        app: hive-clusterimagesets                                                 
        subscription-pause: "false"                                                
    name: hive-clusterimagesets-subscription-stable-0                            
    namespace: open-cluster-management                                           
spec:                                                                          
    channel: open-cluster-management/acm-hive-openshift-releases-chn-0           
    placement:                                                                   
    local: true

Create the object and wait for a while. You can list the new set of ClusterImageSets and see that the new ones belongs to stable channel:

> oc get clusterimagesets img4.9.9-x86-64-appsub  -o jsonpath={.metadata.labels.channel}
stable

After that, you can change from one channel to another, just pausing and unpausing the subscriptions.

ClusterImageSets Refresh Frequency

The refresh frequency is managed by the channel resource. See the following command:

> oc get channel acm-hive-openshift-releases-chn-0 -n open-cluster-management -o json | grep apps.open-cluster-management.io/reconcile-rate
"apps.open-cluster-management.io/reconcile-rate": "low",

In this case, it is configured as low, which means hourly. Different values can be found here.

Creating ClusterImageSets Clusters for Multiple Architectures

Automatically synced ClusterImageSets provide Red Hat OpenShift images for only one architecture. But you can create (manually) your own ClusterImageSets pointing to different architectures.

As a result, an x86 hub cluster deploys ARM clusters, or clusters from any other supported architecture. See this documentation for further information.

Known Issue

By default, RHACM subscribes the list of ClusterImageSets to a fast channel. But the deployed clusters contains RHOCP clusters subscribed to stable channel. This is a known issue/limitation.