Deploying Red Hat OpenShift Operators in a disconnected environment

Deploying a Red Hat OpenShift Operator in an environment with internet access is typically straightforward. However, in industries like cyber security or the military sector, where security concerns often prohibit internet access, the process becomes more complex. In a disconnected or air-gapped environment, internet access is usually restricted or unavailable.

In this article, I demonstrate the process of deploying an operator in a disconnected environment. I use the recent Red Hat OpenShift AI operator for this example, because the use of artificial intelligence is becoming crucial to many environments.

Prerequisites

The steps involve saving all required images to a disk, transferring the disk to the disconnected environment, uploading the OpenShift images to a private registry, and then installing the operator

To proceed, you must have the following requirements in place:

• OpenShift is installed in a disconnected environment
• Object storage is available
• A private registry is deployed in the disconnected environment

Mirroring images from public registries to disk

To complete the first task, you use the oc-mirror plugin, an excellent tool for managing and controlling the volume of data being mirrored.

This section requires an internet connection, because you're fetching images from public registries.

For each operator you want to install in an air-gapped environment, it's essential to carefully review the documentation. This ensures that you mirror all the necessary images and their dependencies.

The OpenShift AI Operator documentation reveals that several other operators need to be installed first. Similarly, the OpenShift Service Mesh has its own set of dependencies. Missing these dependencies won’t necessarily cause the OpenShift AI Operator installation to fail, but it could lead to issues when using it. For example, if you enable the KServe component when deploying the DataScienceCluster instance, you must also have installed Operators for Red Hat OpenShift Serverless and Red Hat OpenShift Service Mesh in order to support KServe (used by the single-model serving platform to serve large models).

The Openshift AI operator (rhods-operator) has these requirements:

• Pipeline operator
• Serverless operator
• Servicemesh operator
  • Jaeger operator
  • Kiali operator

Red Hat Openshift AI needs extra images that can be found in the Git repository RHOAI disconnected install helper, which includes the list of extra images required for Openshift AI 2.11.

The oc-mirror plugin

Before using the oc-mirror plugin, you need to complete some configuration steps. Fortunately, there is comprehensive documentation available to guide you through this process.

Install the oc-mirror plugin

1. To download the oc-mirror CLI plugin, navigate to the downloads page of the OpenShift Cluster Manager. Under the OpenShift disconnected installation tools section, click Download for OpenShift Client (oc) mirror plugin and save the file

2. Extract the archive:

   $ tar xvzf oc-mirror.tar.gz
   $ chmod +x oc-mirror

   Do not rename the oc-mirror file

3. To install the oc-mirror CLI plugin, place the file in your PATH. For example:
   $ sudo mv oc-mirror /usr/local/bin/
4. Verify that the plugin for oc-mirror is installed:
   $ oc mirror help

Configure the oc-mirror credentials

Next, you must configure the oc-mirror credentials. As usual, read the documentation before attempting this.

1. Download your registry.redhat.io pull secret from Red Hat OpenShift Cluster Manager

2. Convert your pull secret in JSON format:

   $ cat ./pull-secret | jq . > <path>/<pull_secret_file_in_json>

3. Copy the JSON file to the .docker directory:

   $ mkdir ~/.docker
   $ cp <path>/<pull_secret_file_in_json> ~/.docker/config.json

Validate the oc-mirror configuration

Validate the oc-mirror configuration:

1. Verify that the plugin for oc-mirror is installed:
   $ oc mirror help
   • Note that there is no dash (-) between oc and mirror.
   • If the command returns help options, then oc-mirror is installed.

2. Log in to registry.redhat.io and look for a credentials are valid message:

   $ podman login registry.redhat.io
   Authenticating with existing credentials for registry.redhat.io
   Existing credentials are valid. Already logged in to registry.redhat.io

Create the oc-mirror ISC (ImageSourceConfiguration) configuration file

The oc-mirror tool uses an ISC configuration file to define what needs to be mirrored. This configuration allows you to specify the catalog source name and the versions of the operators you want to mirror.

To learn how to automate the creation of ISC files, read my article on developers.redhat.com.

Find operator channel and release information

You need to provide the oc-mirror plugin with the operator's channel and version. To get that information, get a list of the available catalogs: 

$ oc mirror list operators --catalogs --version=4.16
Available OpenShift OperatorHub catalogs:
OpenShift 4.16:
registry.redhat.io/redhat/redhat-operator-index:v4.16
registry.redhat.io/redhat/certified-operator-index:v4.16
registry.redhat.io/redhat/community-operator-index:v4.16
registry.redhat.io/redhat/redhat-marketplace-index:v4.16

Find the available packages within the selected catalog

In this scenario, all operators you're mirroring are from the Red Hat operator catalogs.

$ oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.16
NAME                                          DISPLAY NAME                                             DEFAULT CHANNEL
3scale-operator                               Red Hat Integration - 3scale                             threescale-2.14
advanced-cluster-management                   Advanced Cluster Management for Kubernetes               release-2.9
amq-broker-rhel8                              Red Hat Integration - AMQ Broker for RHEL 8 (Multiarch)  7.11.x
amq-online                                    Red Hat Integration - AMQ Online                         stable
amq-streams                                   AMQ Streams                                              stable

Find channels for the selected package

Next, get the channels for the package:

$ oc mirror list operators --catalog=registry.redhat.io/redhat/redhat-operator-index:v4.16 --package=rhods-operator
NAME            DISPLAY NAME          DEFAULT CHANNEL
rhods-operator  Red Hat OpenShift AI  fast
PACKAGE         CHANNEL      HEAD
rhods-operator  alpha        rhods-operator.2.11.0
rhods-operator  beta         rhods-operator.2.4.0
rhods-operator  embedded     rhods-operator.2.7.0
rhods-operator  eus-2.8      rhods-operator.2.8.4
rhods-operator  fast         rhods-operator.2.11.0
rhods-operator  stable       rhods-operator.2.10.0
rhods-operator  stable-2.10  rhods-operator.2.10.0
rhods-operator  stable-2.8   rhods-operator.2.8.4

Find release information in the selected channel

Look in the output for the versions available in the selected channel.

VERSIONS
…
…
2.8.1
1.32.0
2.10.0
2.11.0

Repeat this process for each operator until you've gathered this information for each operator you want to mirror.

Build the oc-mirror ISC configuration file

If this is the first time you're creating an ISC file, use the oc mirror command to create a template.

$ oc mirror init > imageset-config.yaml

Edit the new ISC file and update it using the good catalog and the operators release information you gathered in the previous steps. Here's an example of an ISC configuration file for OpenShift AI 2.11:

$ cat redhat-op-v4.16-config_ai.yaml
kind: ImageSetConfiguration
apiVersion: mirror.openshift.io/v1alpha2
storageConfig:
  local:
    path: /path/to/disk-rh-ai/metadata
mirror:
  operators:
  - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16
    targetCatalog: redhat-catalog-v4.16
    packages:
    - name: "jaeger-product"
      channels:
      - name: "stable"
        minVersion: "1.57.0-7"
    - name: "kiali-ossm"
      channels:
      - name: "stable"
        minVersion: "1.73.8"
    - name: "openshift-pipelines-operator-rh"
      channels:
      - name: "latest"
        minVersion: "1.15.1"
    - name: "rhods-operator"
      channels:
      - name: "fast"
        minVersion: "2.11.0"
    - name: "serverless-operator"
      channels:
      - name: "stable"
        minVersion: "1.33.1"
    - name: "servicemeshoperator"
      channels:
      - name: "stable"
        minVersion: "2.5.2-0"
  additionalImages:   
    - name: quay.io/integreatly/prometheus-blackbox-exporter@sha256:35b9d2c1002201723b7f7a9f54e9406b2ec4b5b0f73d114f47c70e15956103b5
    - name: quay.io/modh/caikit-nlp@sha256:0cde6c26e02ec398aea959a1a1bcdc615b86821adb41989e81d03de01124545c
    - name: quay.io/modh/caikit-tgis-serving@sha256:4e907ce35a3767f5be2f3175a1854e8d4456c43b78cf3df4305bceabcbf0d6e2
…
…
…

Finally, create a directory for the image's TAR file, and then run the command:

$ mkdir disk-rh-ai
$ oc mirror --verbose 3 --config=redhat-op-v4.16-config_ai.yaml file://disk-rh-ai

Once complete, the TAR file is in the disk-rh-ai directory you created:

$ ls -ltr disk-rh-ai/
drwxr-xr-x. 3 admin admin           17 Aug  1 10:56 oc-mirror-workspace
-rw-r--r--. 1 admin admin 176802807808 Aug  1 11:29 mirror_seq1_000000.tar
drwxr-x---. 3 admin admin           21 Aug  1 11:29 metadata

Move the TAR file to the disconnected environment

The operational procedures and security requirements can vary significantly from one organization to another. For example, in certain highly restricted environments, you must copy the TAR file to a portable disk, which you take to your disconnected environment's security team for review.

Mirroring the operator from disk to private registry

Once the data has been moved to the disconnected environment, you can begin pushing the images in your private registry. Because you've used the oc-mirror plugin to push all necessary container images, you must configure the tools accordingly, with the credential information pointing to your private registry instead of the Red Hat public registry.

Here is an example of the oc-mirror credential configuration for a private registry:

$ cat ~/.docker/config.json 
{
  "auths": {
    "quaylab.redhat.com:8443": {
      "auth": "aW5pdDpVUjiAYzJiOGcxTVc01djQcGh6PQ==",
      "email": "ebeaudoi@example.com"
    }
}

Upload the container’s images to your private registry

Create a directory to copy the TAR file, and to serve as the destination for “oc mirror” output:

$ mkdir disk-rh-ai
$ cp mirror_seq1_000000.tar disk-rh-ai/
$ cd disk-rh-ai

Execute the oc mirror command to upload the container’s images to your private registry

$ oc mirror --verbose 3 \
--from=./mirror_seq1_000000.tar \
docker://quaylab.redhat.com:8443/redhat-v416

Once complete, you get the following directories with similar files:

$ ls -ltr oc-mirror-workspace/results-1722527218/
total 60
drwxr-xr-x. 2 admin admin     6 Aug  1 11:46 charts
drwxr-xr-x. 2 admin admin     6 Aug  1 12:47 release-signatures
-rw-r--r--. 1 admin admin 50289 Aug  1 12:48 mapping.txt
-rwxr-xr-x. 1 admin admin   264 Aug  1 12:48 catalogSource-cs-redhat-catalog-v4-16.yaml
-rwxr-xr-x. 1 admin admin  2384 Aug  1 12:48 imageContentSourcePolicy.yaml

Configure the Openshift catalog

Keeping the Openshift default catalogs in a disconnected environment would just trigger unnecessary errors, so disable the default OperatorHub catalog sources:

$ oc patch OperatorHub cluster --type json -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

Create a new catalog referring to your private registry using the YAML files generated by the oc-mirror plugin output.

Log in with cluster-admin privileges

You must be logged in to your OpenShift instance as a user with cluster-admin privileges. To confirm:

$ oc login -u admin -p redhat https://api.quaylab.redhat.com:6443
$ oc whoami
admin

Create the ICSP file using Openshift CLI

Now you can create the ICSP file:

$ oc apply -f oc-mirror-workspace/results-1722527218/imageContentSourcePolicy.yaml 
…
imagecontentsourcepolicy.operator.openshift.io/generic-0 created
imagecontentsourcepolicy.operator.openshift.io/operator-0 created

Create the Red Hat catalog using the Openshift CLI

Next, create the Red Hat catalog:

$ oc apply -f oc-mirror-workspace/results-1722527218/catalogSource-cs-redhat-catalog-v4-16.yaml 
…
catalogsource.operators.coreos.com/cs-redhat-catalog-v4-16 created

Verify that you can see the catalog in the mirrored operators in Operators > OperatorHub:

Configure the OpenShift internal registry

An OpenShift internal registry is a dependency for OpenShift Data Science and OpenShift AI in releases prior to 2.10. For a test cluster, you can use emptyDir. Do not use emptyDir for a production cluster.

$ oc patch configs.imageregistry.operator.openshift.io cluster \
--type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}' \
config.imageregistry.operator.openshift.io/cluster 
$ oc patch configs.imageregistry.operator.openshift.io cluster \
--type merge --patch '{"spec":{"managementState":"Managed"}}' \ 
config.imageregistry.operator.openshift.io/cluster

Install the AI operators

I suggest using the OpenShift UI to install the operators. Once you have the OpenShift cluster UI open, follow these steps:

1. Click on Operators > OperatorHub
2. Select the operator to install
3. Keep the default settings and click Install

Repeat this process for each operator:

• Kiali operator
• Jaeger operator
• Pipeline operator
• Serverless operator
• Servicemesh operator
• Openshift AI

Create the RHOAI instance

After the OpenShift AI Operator is successfully installed, you must create a DataScienceCluster instance.

1. In the web console, click Operators > Installed Operators, and then select Red Hat OpenShift AI Operator
2. Click the Data Science Cluster tab

3. Click Create DataScienceCluster 

   

4. In the UI, select YAML view to see the installed components. You have two options:

   • Managed: The Operator actively manages the component, installs it, and tries to keep it active. The Operator upgrades the component only if it is safe to do so
   • Removed: The Operator actively manages the component but does not install it. If the component is already installed, the Operator tries to remove it
   
5. Click the Create button to continue

Access the Red Hat AI dashboard

At this point,the Openshift AI Data Science cluster instance has been successfully deployed. The OpenShift AI dashboard has its own UI.

1. To access the OpenShift AI dashboard, click the Grid icon application launcher at the top of the screen

   
2. Right-click on Red Hat OpenShift AI and copy the URL of your OpenShift AI instance

3. Paste the URL into a web browser to navigate to the dashboard

   

Conclusion

Working in a disconnected environment presents significant challenges and adds complexity to the application development process. This mode of operation requires enhanced control over how your applications interact with the internet. Using the right tools is crucial for managing data efficiently and minimizing your workload. In this blog, I've demonstrated how to use the oc-mirror plugin, which I believe is the best approach for this scenario. It provides enhanced control over the volume of data you need to handle, streamlines the process, and improves efficiency.

If you're interested in knowing more about Red Hat AI, click here. If you want to know more about Openshift in disconnected environment, click here.