Red Hat Advanced Cluster Management (RHACM) is a solution that allows you to manage Kubernetes/OpenShift clusters and infrastructures. It includes OpenShift Hive, an operator that runs as a service based on OpenShift. The Hive service can be used to provision and perform initial configuration of OpenShift clusters. Hive enables deployments with different cloud providers, such as AWS, Azure, or Google Cloud Platform, as well as baremetal infrastructure.
OpenShift Hive provides a custom resource called ClusterImageSet, which points to different available OpenShift versions. You can easily check which available OpenShift versions are available with the following command:
> oc get clusterimagesets.hive.openshift.io
NAME RELEASE
img4.10.3-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.10.3-x86_64
img4.10.4-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.10.4-x86_64
img4.10.5-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.10.5-x86_64
img4.6.1-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.1-x86_64
img4.6.12-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.12-x86_64
img4.6.13-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.13-x86_64
img4.6.15-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.15-x86_64
img4.6.16-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.16-x86_64
img4.6.17-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.17-x86_64
img4.6.18-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.18-x86_64
img4.6.19-x86-64-appsub quay.io/openshift-release-dev/ocp-release:4.6.19-x86_64
.....
The oc CLI provides the full list of available images. However, when using RHACM console, this list is automatically filtered to show only the latest versions. This encourages the creation of clusters with the latest release images but does not prevent use of previous versions. See the following command:
> oc get clusterimagesets.hive.openshift.io img4.10.10-x86-64-appsub -o jsonpath={.metadata.labels.visible}
false
> oc get clusterimagesets.hive.openshift.io img4.10.15-x86-64-appsub -o jsonpath={.metadata.labels.visible}
true
Subscribing ClusterImageSet
The list of available images is automatically synced using a combination of subscription and channel resources. See the following information about subscription manifests (simplified for better reading):
> oc get subscription hive-clusterimagesets-subscription-fast-0 -n open-cluster-management -oyaml
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
annotations:
apps.open-cluster-management.io/git-branch: release-2.5
apps.open-cluster-management.io/git-path: clusterImageSets/fast
creationTimestamp: "2022-03-29T12:39:46Z"
labels:
subscription-pause: "false"
name: hive-clusterimagesets-subscription-fast-0
namespace: open-cluster-management
spec:
channel: open-cluster-management/acm-hive-openshift-releases-chn-0
placement:
local: true
status:
lastUpdateTime: "2022-05-24T22:20:38Z"
message: Active
phase: Subscribed
statuses:
/:
packages:
acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.1-x86-64-appsub:
lastUpdateTime: "2022-04-17T14:43:26Z"
phase: Subscribed
acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.3-x86-64-appsub:
lastUpdateTime: "2022-04-17T14:43:25Z"
phase: Subscribed
acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.4-x86-64-appsub:
lastUpdateTime: "2022-04-17T14:43:27Z"
phase: Subscribed
acm-hive-openshift-releases-chn-0-ClusterImageSet-img4.6.6-x86-64-appsub:
See the following important information we can check for in the subscription:
- The subscription uses a fast channel and the release
2.5branch, which is the current version of RHACM in the example. - The subscription is active (not paused).
- The
lastUpdateTimeshows the last update time for the subscription. - There is a list of all the
ClusterImageSetreceived.
The spec.channel points to the corresponding channel resource, which keeps the communication with the repository containing all the ClusterImageSets. In this case, the subscription is using the open-cluster-management/acm-hive-openshift-releases-chn-0 RHACM channel.
See the following to get more info about the RHACM channel:
> oc get channel acm-hive-openshift-releases-chn-0 -n open-cluster-management -oyaml
apiVersion: apps.open-cluster-management.io/v1
kind: Channel
metadata:
annotations:
apps.open-cluster-management.io/reconcile-rate: low
creationTimestamp: "2022-03-29T12:39:46Z"
name: acm-hive-openshift-releases-chn-0
namespace: open-cluster-management
spec:
pathname: https://github.com/stolostron/acm-hive-openshift-releases.git
type: Git
The channel points to a Git repository. This repository is maintained and updated by Red Hat with the different available OCP versions in the way of ClusterImageSets. See that the subscription uses this channel, selecting the git-repo and git-branch:
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
annotations:
apps.open-cluster-management.io/git-branch: release-2.5
apps.open-cluster-management.io/git-path: clusterImageSets/fast
With this mechanism, RHACM is updated with all the available versions of RHOCP.
Combining the Git URL configured by the channel, the git-branch and the git-path from the subscription, you can browse to the repo, where the different ClusterImageSets are downloaded.
Changing the Channel to Sync
RHACM comes with the fast channel installed by default. This can be changed by following next steps. If you are just using RHACM, by default, all the resources should be managed into the open-cluster-management Namespace. See the following procedure:
- Pause the fast channel subscription. Run the following command:
> oc -n open-cluster-management patch appsub hive-clusterimagesets-subscription-fast-0 --type='json' -p='[{"op":"replace","path": "/metadata/labels/subscription-pause","value":"true"}]'
subscription.apps.open-cluster-management.io/hive-clusterimagesets-subscription-fast-0 patched
- Delete all the current available
ClusterImageSets. These objects only contain references to download images, so you are not really deleting anything critical. The resources we are deleting belong to the fast channel. Ensure you did not manually createClusterImageSets; these will be also deleted and you may want to keep them:
> oc delete clusterimageset --all
clusterimageset.hive.openshift.io "img4.10.10-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.11-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.12-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.13-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.14-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.15-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.3-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.4-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.5-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.6-x86-64-appsub" deleted
clusterimageset.hive.openshift.io "img4.10.8-x86-64-appsub" deleted
- Add a new subscription for stable channel.
Best practice: Get the Manifest from your current subscription. Just clean the usual extra data created by OpenShift/Kubernetes.
- Finally, change the
git-patch, ensure it is not paused, and change the name forstable-0. See the following example based on the preinstalled fast subscription:
apiVersion: apps.open-cluster-management.io/v1
kind: Subscription
metadata:
annotations:
apps.open-cluster-management.io/git-branch: release-2.5
apps.open-cluster-management.io/git-path: clusterImageSets/stable
labels:
app: hive-clusterimagesets
subscription-pause: "false"
name: hive-clusterimagesets-subscription-stable-0
namespace: open-cluster-management
spec:
channel: open-cluster-management/acm-hive-openshift-releases-chn-0
placement:
local: true
- Create the object and wait for a while. You can list the new set of
ClusterImageSetsand see that the new ones belongs tostablechannel:
> oc get clusterimagesets img4.9.9-x86-64-appsub -o jsonpath={.metadata.labels.channel}
stable
After that, you can change from one channel to another, just pausing and unpausing the subscriptions.
ClusterImageSets Refresh Frequency
The refresh frequency is managed by the channel resource. See the following command:
> oc get channel acm-hive-openshift-releases-chn-0 -n open-cluster-management -o json | grep apps.open-cluster-management.io/reconcile-rate
"apps.open-cluster-management.io/reconcile-rate": "low",
In this case, it is configured as low, which means hourly. Different values can be found here.
Creating ClusterImageSets Clusters for Multiple Architectures
Automatically synced ClusterImageSets provide Red Hat OpenShift images for only one architecture. But you can create (manually) your own ClusterImageSets pointing to different architectures.
As a result, an x86 hub cluster deploys ARM clusters, or clusters from any other supported architecture. See this documentation for further information.
Known Issue
By default, RHACM subscribes the list of ClusterImageSets to a fast channel. But the deployed clusters contains RHOCP clusters subscribed to stable channel. This is a known issue/limitation.
