Gathering security data for container images using the Pyxis API

In the previous post, we covered how to use the Red Hat Security Data API to collect useful security information about CVEs and Red Hat products programmatically. 

In this post, we’ll look at how to collect security data for container images by using the Pyxis API. As before, we will be addressing real world use cases and concerns programmatically. Each of the examples used below can be easily modified to address your own needs.

Use cases

How do I list RPMs in my container image? Are there any available updates?

As we saw in the previous post, every Red Hat container image contains RPMs. These packages could be related to the Universal Base Image (UBI) image that has been used as a base image, or could be installed during the container build process. To get a package manifest for the selected container, you can use Pyxis endpoints. Let’s take an example container image:
Red Hat Universal Base Image 8
Architecture: amd64 Tag: 8.4-206

The Pyxis API url is: https://catalog.redhat.com/api/containers/v1
(To make queries easier to read, let’s assign the Pyxis API url to variable $PYXIS_URL.)
First, you need to find the repository url for your image. To do that, we will use the podman command with the inspect parameter to query details about the image.  For example:

podman inspect your_image | jq '.[].Labels.url'
"https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.4-209"

With this information, we now use the Pyxis repository endpoint to gather additional image metadata:

curl -s "$PYXIS_URL/repositories/registry/registry.access.redhat.com/repository/ubi8/images" | jq .

There is a lot of interesting information about all the versions of UBI 8 image in this repository and related updates. There are also links to the package manifests. To gather this information, you can run this query:

curl -s "$PYXIS_URL/repositories/registry/registry.access.redhat.com/repository/ubi8/images" | jq -r '.data[] | select(.architecture =="amd64") | {"version": .brew.nvra, "id": ._id, "link": ._links.rpm_manifest.href}'

As an output, you will get a list of all image versions and related images id and rpm-manifests. For image with tag 8.4-209, it will be:

…
{
  "version": "ubi8-container-8.4-206.amd64",
  "id": "60d221c03d73db4108ade7eb",
  "link": "/v1/images/id/60d221c03d73db4108ade7eb/rpm-manifest"
}
...

Now you can gather information about the RPMs in one individual image from that list:

curl -s "$PYXIS_URL/images/id/60d221c03d73db4108ade7eb/rpm-manifest"

(Note that the /v1/ is duplicated in the Pyxis URL and at the start of the RPM manifest link  generated. When we continue here, we strip off the duplicate as shown in the next command.)

To make the output formatted and just list RPMs you may use jq parser like this:

curl -s "$PYXIS_URL/images/id/610973214841f140fd7b5bf0/rpm-manifest" | \
jq -r '.rpms[]? | .nvra' | sort

From this output, we see that our UBI8 8.4-209 container ships the systemd-239-45.el8_4.1.x86_64 package. Let’s now combine this information with a use case from our previous blog post, where we analyzed the CVE related to the systemd package.

To check available updates to packages in our image we can use another Pyxis endpoint: vulnerabilities

Since we already collected information about our image id (60d221c03d73db4108ade7eb), we need to just run the query like this:

curl -s "$PYXIS_URL/images/id/60d221c03d73db4108ade7eb/vulnerabilities"

In the output, you will see the list of CVEs which are not addressed yet in your image and information about the advisory with the fix:

curl -s "$PYXIS_URL/images/id/60d221c03d73db4108ade7eb/vulnerabilities" | jq -r '.data[] | "\(.cve_id) | \(.advisory_type)-\(.advisory_id)"'
CVE-2021-33910 | RHSA-2021:2717
CVE-2021-27218 | RHSA-2021:3058

We can also tweak one of the recent Pyxis queries and check details about the latest available UBI 8 image and verify if that image is still affected or not.

By adding optional parameters sort_by= and page_size= to the queried Pyxis endpoint, we can get a list of images with their id and corresponding rpm-manifests sorted by creation date:

curl -s "$PYXIS_URL/repositories/registry/registry.access.redhat.com/repository/ubi8/images?sort_by=creation_date&page_size=500 " | jq -r '.data [] | select(.architecture =="amd64") | \
{"version": .brew.nvra, "id": ._id, "link": ._links.rpm_manifest.href}

From this query we will know that the latest UBI 8 image has tag 8.4-211:

  "version": "ubi8-container-8.4-211.amd64",
  "id": "6140d134702c563e892d3576",
  "link": "/v1/images/id/6140d134702c563e892d3576/rpm-manifest"

By querying the vulnerabilities Pyxis endpoint for the newest image, we can confirm that this image is not known to be vulnerable anymore (there is no output):

curl -s "$PYXIS_URL/images/id/6140d134702c563e892d3576/vulnerabilities" |jq -r '.data[] | "\(.cve_id) | \(.advisory_type)-\(.advisory_id)"'

Based on the collected information, we know that the newest version of the UBI image is ubi8-container-8.4-211 and is not vulnerable to the CVE-2021-33910 anymore. We can also check what systemd package version is shipped in that UBI 8 version:

curl -s "$PYXIS_URL/images/id/6140d134702c563e892d3576/rpm-manifest" | jq -r '.rpms[]? | .nvra' | sort -u
systemd-239-45.el8_4.3.x86_64

It means that our UBI 8 8.4-209 container image ships systemd-239-45.el8_4.1.x86_64 and is vulnerable to CVE-2021-33910, whereas the newest UBI 8 container image ships already fixed version of the systemd package (systemd-239-45.el8_4.3.x86_64).

Note:

• Some products use the “latest” tag for images to show the latest released image. Others do not—for example, Red Hat Quay. In this use case, we didn’t rely on the “latest” tag, but if you know that the “latest” tag is still available in the examined image repository, then you may use repository Pyxis endpoint with tag option to list the latest available image:

curl -s curl -s "$PYXIS_URL/repositories/registry/registry.access.redhat.com/repository/ubi8/tag/latest" | jq -r '.data [] | select(.architecture =="amd64") | [ .brew.nvra, ._id, ._links.rpm_manifest.href ]'

• It is always recommended to use the newest available version of the image. In our example, we found that ubi8-container-8.4-206.amd64 has a vulnerable systemd package (CVE-2021-33910).
  
  That vulnerability was fixed in systemd-239-45.el8_4.2 delivered by RHSA-2021:2717 advisory. But we also verified that in the latest delivered version of the UBI 8 image, there is already a newer version of the systemd-239-45.el8_4.3 package that was shipped to address CVE-2021-33910. This means that there were other updates delivered in other advisories after the RHSA-2021:2717.

How do I gather information about installed RPMs in a selected container image from a specified product?

To address this question, we need to list all the repositories within container images for a selected product. To list all Red Hat products, you can run this query using the Pyxis API:

curl -s https://catalog.redhat.com/api/containers/v1/product-listings?filter=vendor_label==redhat | jq -rc '.data[] | [.name, ._id]' | sort

The output is a representation of the Red Hat certified software page where products and related container images are categorized into four categories. You will notice that some products are displayed twice with different Product ID numbers—for example, Red Hat Quay:

["Red Hat Quay","5ec53f9d535cb70ab8c02991"]
["Red Hat Quay","601a85c2895df448347e71fa"]

This is because Red Hat Quay belongs to “OpenShift operators” and “Containerized products” categories. But the repositories comparison for the Product IDs shows that both are pointing to the same repositories with the same container images.

To list all the repositories for a selected product and get a link to stored images, run:

curl -s "$PYXIS_URL/product-listings/id/5ec53f9d535cb70ab8c02991/repositories" | \ 
jq -r '.data[] | ._links.images.href'

(Where PYXIS_URL=https://catalog.redhat.com/api/containers/v1)

/v1/repositories/registry/registry.access.redhat.com/repository/quay/clair-rhel8/images
/v1/repositories/registry/registry.access.redhat.com/repository/quay/quay-builder-qemu-rhcos-rhel8/images
/v1/repositories/registry/registry.access.redhat.com/repository/quay/quay-rhel8/images
/v1/repositories/registry/registry.access.redhat.com/repository/quay/quay-operator-rhel8/images
/v1/repositories/registry/registry.access.redhat.com/repository/quay/quay-builder-rhel8/images

The returned list is a representation of what is visible on the Red Hat Container Catalog page, where:
/v1/repositories/registry/registry.access.redhat.com/repository/quay/quay-rhel8/images

is a representation of:
https://catalog.redhat.com/software/containers/quay/quay-rhel8/600e03aadd19c7786c43ae49

To programmatically list all images in a specific repository and gather information about each image, you can run this query (sorted by creation date and limiting to the amd64 arch):

curl -s "$PYXIS_URL/repositories/registry/registry.access.redhat.com/repository/quay/quay-rhel8/images?sort_by=creation_date&page_size=500" | jq -r '.data[] | select(.architecture == "amd64")'

In the collected data, you will find useful information such as:

• Image build date.

• Image content sets.

• Image metadata in labels section.

• Image environment variables.

• Image health index grade.

• Assigned CPE (necessary for OVAL data comparison).

• Links to RPM manifests.

We have information about all the repositories used by our products and list all images, including information about their RPM manifests. Like in the previous use case, we can now query the Pyxis rpm-manifest endpoint and gather information about RPMs in every product image. For example, to get RPM manifest for the latest quay/quay-rhel8 image we can run:

curl -s "$PYXIS_URL/images/id/6143bb4bd095e30ed5724b92/rpm-manifest" | jq -r '.rpms[]? | .nvra' | sort

In summary, it’s now possible to programmatically gather RPM manifests from all images in selected products. You can also check what the selected package version is in the latest image and compare this to the package version that is installed in the older version (for example, what is currently used in your deployment). 

It is important to mention that the available image RPM manifests only cover the default shipped packages in the image. For example, if you use Red Hat UBI 8 in your deployment , and you install some additional packages during the build process, these packages won’t be listed in the UBI 8 image RPM manifest.

In the Red Hat OpenShift Container Platform cluster, I use a specific container image in my project. How can I check the security health index for it?

As a user of Red Hat OpenShift Container Platform, you can list all available images by executing the following command: oc get images

In the output, there will be a list of the image IDs (repositories and the sha digest). Learn more about containers, images and imagestreams here. 

To check the selected image, you first need to obtain an official pull secret since all images are shipped from protected repositories. As an OpenShift Container Platform cluster administrator, you can extract the cluster pull secret with oc extract secret/pull-secret --namespace openshift-config, storing the file in the current directory.

If you’re not logged into an OpenShift cluster, you can also obtain a pull secret from the Hybrid Cloud Console. 

As an example, let’s select the ubi8/python-38 container image, which will be displayed in the output from the `oc get images` command:

registry.redhat.io/ubi8/python-38@sha256:43b037584dac3425f845faab4f3575a9058c431219ffbf735f937b263713f2d9

Now, it’s necessary to check image details (metadata). That can be achieved by running the following command:

oc image info --registry-config=.dockerconfigjson registry.redhat.io/ubi8/python-38@sha256:43b037584dac3425f845faab4f3575a9058c431219ffbf735f937b263713f2d9

where dockerconfigjson is the pull secret gathered in the earlier step.

In the output, the Labels section contains the information about the image registry repository url. For example, for the above image it is:

url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/python-38/images/1-71

Based on this, we can list all the available images in that repository and check a specific image health index. We can also use information about the repository and select a specific image tag (in this example 1-71) and gather image metadata: curl -s "$PYXIS_URL/repositories/registry/registry.access.redhat.com/repository/ubi8/python-38/tag/1-71"

Based on the collected image id (6141b1e54841f1221d0b1750), we can check the image health index and see if there are any pending CVEs to address:

curl -s "$PYXIS_URL/images/id/6141b1e54841f1221d0b1750/vulnerabilities" | jq -r '.data[] | "\(.cve_id);\(.advisory_type)-\(.advisory_id)"'
CVE-2021-3672;RHSA-2021:3666
CVE-2021-23343;RHSA-2021:3666
CVE-2021-22931;RHSA-2021:3666
CVE-2021-22930;RHSA-2021:3666
...

To collect similar health index information for the OpenShift Container Platform core images, you can list all images in the specific OpenShift Container Platform release with the following command (you need to pull secret like before):

oc adm release info 4.9.0 --registry-config=.dockerconfigjson

For the output, you will get a list of image IDs which you can examine and check the associated health indexes accordingly.

Conclusion

The Security Data and Pyxis APIs are powerful services that can be used to gather useful security related information programmatically. You can regularly check the container health index for selected important images, collect information about the installed packages updates, verify CVEs and validate the impact to your products.

By gathering information about the Common Platform Enumeration (CPE) associated with an image repository, you can query the Security Data API to collect information about fixes for a specific CPE. The repository CPE details can also be used to determine the correct OVAL data, associated with the repositories. OVAL data is needed to perform additional security scanning for vulnerabilities with OpenSCAP tool, for example. More information can be found in this Enable SysAdmin post or this Red Hat Blog post.

A lot of people use scanning products (security scanners) for their systems/containers and the results sometimes are difficult to understand and contain a lot of false positives findings. The reports prepared for Red Hat products can be verified as shown in all example use cases. To address this problem Red Hat has started a vulnerability certification for scanners program.

Learn more about how Red Hat can help with security by visiting our main security topic page and Red Hat Blog security channel.