Quay OCI Artifact Support for Helm Charts

This blog has been updated as of 4/11/22

Since the early days of Kubernetes, other than the API manifests, the primary supporting artifact has been a container image. Container images are stored and accessed by Kubernetes from container registries. These registries were originally designed to support the Docker image format; however, to promote additional runtimes other than Docker, the Open Container Initiative (OCI) was created to provide a standardization surrounding container runtimes and image formats. Most container registries support the OCI standardization, which is based on the Docker image manifest V2, Schema 2 format. 

As the Kubernetes platform has evolved, a variety of artifacts aside from container images have emerged that support not only individual applications, but also the platform as a whole. These range from Open Policy Agent (OPA) policies, security, and governance to Helm charts and Operators to aid in application deployment. One of the challenges of these supplementary components is how and where they should be stored. While each can make use of its own implementation, a common artifact store and/or format would be better. Fortunately, the OCI specification can be used for this, and since container registries can implement the OCI specification, they can provide a solution.

Red Hat Quay is a private container registry that stores container images as well as an entire ecosystem of supporting tooling to aid in managing containers. The release of Red Hat Quay 3.4 introduced a new Technology Preview feature that enables using OCI-based artifacts and, specifically, Helm Charts. With the release of Red Hat Quay 3.6, this feature was graduated to full supportability. The remainder of this discussion will introduce how OCI support for Helm charts in Quay can be used.

Support for Helm charts within Quay has existed initially through the concept of an Application Registry. This feature was the first attempt to provide storage for other mostly text-based artifact types beyond container images before OCI introduced the concept of media types. The long-term approach moving forward is to use the OCI format and the Application Registry component will be removed in future versions of Red Hat Quay.

Deploying and Configuring Quay

Support for Helm charts and OCI artifacts is available in both the public-hosted quay.io  service as well as the self-hosted Red Hat Quay. The easiest method to deploy the self hosted Red Hat Quay is in OpenShift using the Quay Operator (Quay can also be deployed outside a Kubernetes-based control plane on physical or virtual hosts, but the setup and configuration are outside the scope of this article).

Make sure that you are logged into an OpenShift environment and create a new project called quay:

$ oc new-project quay

The next step is to deploy the Quay Operator from OperatorHub. The operator features a “batteries-included” configuration that enables a seamless deployment to get users up and running quickly with minimal effort. Customizations can be specified to enable a more fine grained level of configuration if desired. Consult the Product Documentation to review the available parameters along with any prerequisite steps that might be required. 

Once the operator has been installed and is running, create the QuayRegistry custom resource and reference the secret containing the config bundle previously created: 

oc create -n quay -f - << EOF
apiVersion: quay.redhat.com/v1
kind: QuayRegistry
metadata:
  name: quay
  namespace: quay
spec: {}
EOF

The URL of the provisioned instance can be found in the registryEndpoint within status field of the QuayRegistry custom resource or by running the following command:

oc get quayregistry -n quay quay -o jsonpath=’{ .status.registryEndpoint }’

Navigate to the URL returned and create an account by clicking the Create Account link.

Enter a username, email address and password and once complete, you will be ready to use Quay to store Helm charts in the OCI format.

Helm OCI Support and Quay

Helm, as a graduated project of the Cloud Native Computing Foundation (CNCF), has become the de facto package manager for Kubernetes as it simplifies how applications are packaged and deployed. Helm uses a packaging format called Charts which contain the Kubernetes resources representing an application. Charts can be made available for general distribution and consumption in repositories. A Helm repository is an HTTP server that serves an index.yaml metadata file and optionally a set of packaged charts. Beginning with Helm version 3, support was made available for distributing charts in OCI registries as an alternative to a traditional repository. To demonstrate how Quay can be used as a registry for Helm charts, an existing chart from a Helm repository will be used to showcase the interaction with OCI registries for chart developers and users 

First, we will download an existing chart from a repository locally to mimic the workflow a chart maintainer or end user would follow to publish via Quay. While there are countless charts and repositories in the community, let's use the etherpad chart from the Red Community of Practice (CoP) repository. Etherpad is an online collaboration tool that enables editing of documents in real time.

Ensure you have the Helm CLI (version 3.8.0+) command line tool and add the repository by executing the following command:

$ helm repo add redhat-cop https://redhat-cop.github.io/helm-charts

Once the repository has been added, obtain the latest metadata:

$ helm repo update

Download the chart locally which will place the packaged chart in the current directory:

$ helm pull redhat-cop/etherpad --version=0.0.4

Starting with Helm 3.8.0, OCI support became fully supported after a lengthy period of time as an “experimental” feature. For users with older versions of the Helm client installed, they will need to set the  environment variable HELM_EXPERIMENTAL_OCI=1:

Prior to publishing a Helm chart to Quay, an Organization must be first available for which the chart will be contained within.Open the browser again, navigate to Quay, and create a new Quay organization to store Helm charts by selecting the + button on the navigation bar and then choosing New Organization. 

Enter helm as the organization name and then click Create Organization.

At this point, the Helm can be used to integrate with Quay.

Interacting With Helm and Quay

Akin to interacting with Quay for traditional container images, we must provide credentials in order to push and pull charts. Communication between the Helm client and Quay is facilitated over HTTPS, and as of Helm 3.8, support is only available for registries communicating over HTTPS with trusted certificates. In addition, the operating system must trust the certificates exposed by the registry. Support in future Helm releases will allow for communicating with remote registries insecurely. With that in mind, make sure that your operating system has been configured to trust the certificates used by Quay. 

Next, login to the remote using the helm registry subcommand with your Quay credentials:

$ helm registry login <QUAY_HOSTNAME>

Username: <username>
Password: <password>

Login succeeded

Push the chart to Quay by specifying the packaged chart and the location within Quay. Notice that OCI registries are denoted by the oci:// protocol in the destination. The name of the repository and tag will match the contents chart.

$ helm push etherpad-0.0.4.tgz oci://<QUAY_HOSTNAME>/helm
Pushed: <QUAY_HOSTNAME>/helm/etherpad:0.0.4
Digest: sha256:74420a94f26201498cc0cd7356952a208391a30ac9770dd36092de97df594d3b

The QUAY_HOSTNAME should be replaced with the hostname for Quay with the value obtained in the oc get quayregistry -n quay quay -o jsonpath=’{ .status.registryEndpoint }’ command used previously:

You can use the Quay web console to see the newly pushed chart in the registry, including listing the available tags on the tags page:

To confirm that Quay is properly serving the chart contents, delete the chart from the local file system:

$ 
$ rm -f etherpad-0.0.4.tgz

This is how a maintainer would publish their Helm charts to Quay as an OCI image. Now let’s look at how users would consume this chart.

Charts located in OCI registries can either be downloaded to the local machine or installed directly from the registry itself. To download a chart to the local machine, use the helm pull command:

$ helm pull oci://<QUAY_HOSTNAME>/helm/etherpad --version=0.0.4

Pulled: <QUAY_HOSTNAME>/helm/etherpad:0.0.4
Digest: sha256:74420a94f26201498cc0cd7356952a208391a30ac9770dd36092de97df594d3b

The chart can then be installed using the helm install command:

$ helm install etherpad-0.0.4.tgz

Alternatively, charts can be installed directly from OCI registries without needing to download locally. Use the helm install command and reference the location within Quay:

$ helm install oci://<QUAY_HOSTNAME>/helm/etherpad --version=0.0.4

As the Kubernetes ecosystem continues to expand, there will be a need to holistically manage the array of supporting components in a centralized manner. The inclusion of OCI-based artifacts, and specifically Helm Charts, is the first step for how the Quay registry can be used to support these workloads along with existing traditional image based content. By default, only Helm charts, cosign (artifact signing), and ztsd based compression are the enabled OCI types enabled. Additional types can be included by customizing the deployment of Quay.