Scanning container image vulnerabilities with Clair

What are container image vulnerabilities?

When Red Hat started releasing products as container images, we decided we needed to build them the same way they were built upstream, which is by building them from Dockerfiles. This means we build fewer RPMs than we used to, and instead we pull in software dependencies from outside of the RPM / DNF ecosystem. It took us some time to get our security data to the same standard as it was for RPM, but that time has arrived, and you can now consume security data for Red Hat container images the same way you do for Red Hat RPMs.

Red Hat named a Leader in the 2023 Gartner® Magic Quadrant™

Red Hat was positioned highest for ability to execute and furthest for completeness of vision in the Gartner 2023 Magic Quadrant for Container Management.

Read the report

How is this different from the existing security content?

When we build container images at Red Hat we write the Dockerfile used to build them into the container itself, and then we release the container via the Container Catalog. Identifying information is written into the Dockerfile during the build process, and you can use that information to compare it with security data we publish about images.

For Red Hat RPM content we publish OVAL files which identify RPMs with security patches in them. OVAL also encapsulates rules for how to scan an operating system for vulnerabilities. In other words, OVAL provides both the data required to scan a system, and the details about how to scan the system. We found that users often want to scan their entire infrastructure — including other operating systems and container registries — with a security scanner that aggregates data from multiple sources.

This means that it makes sense to separate the security data from the rules on how to scan the system, enabling our security scanner partners to consume the data more easily and also providing Red Hat flexibility in how we provide the data.

Data for container image vulnerabilities can be found in the Red Hat CVE Database which most users access via the web interface. The data backing the web interface can be found in a file called cvemap.xml as well as the Security Data API.

Clair is an open source project that provides a tool to monitor the security of your containers through the static analysis of vulnerabilities in container images. Clair has been improved to make use of the data in cvemap.xml, so you can now use it to scan Red Hat Container Images for vulnerabilities — not just in RPMs, but also in other non-RPM content.

Technical details

Support for scanning container images has been added to Clair 4.4.2 via this pull request in Clair Core. Clair is used by quay.io, Red Hat Quay, and the Red Hat Container Catalog (registry.redhat.io) via the Container Health Index to track and report vulnerabilities affecting container images. Until now, Clair has only matched Red Hat vulnerability data with RPMs found in containers, but with the release of Clair 4.4.2, it can match container images as well.

How does it work?

Clair scans the container file content, reads the Dockerfile contents, and extracts the “name” label. It also downloads cvemap.xml file, parses the container image names and fixed-in versions, and then matches it against the “name” label from the Dockerfile.

Sometimes the two names don’t exactly match when they should, so in those cases we provide a mapping file which maps the container image name label with its image name as found in the CVE database.

The reason Clair uses cvemap.xml over the Security Data API is twofold. First, the architecture of Clair is such that it is preferable to consume a flat file containing all the data at once and update it periodically. Second, Clair is often used in offline environments without direct access to the internet. In those environments it is easier to mirror a small number of files into the environment than mirror a whole API service. 

Version matching

To avoid false positives when you’re using a patched container image in one product stream, but a later product stream for the same product patches the vulnerability, we constrain matching to a minor version.

For example, the recent Log4j Critical vulnerability CVE-2021-44228. That issue was addressed in both the following container image versions for the “openshift4/ose-metering-hive” container image, as demonstrated by this section of the cvemap.xml.

<AffectedRelease cpe="cpe:/a:redhat:openshift:4.6::el8">
   <ProductName>Red Hat OpenShift Container Platform 4.6</ProductName>
   <ReleaseDate>2021-12-16T00:00:00</ReleaseDate>
   <Advisory type="RHSA"
      url="https://access.redhat.com/errata/RHSA-2021:5106">RHSA-2021:5106
   </Advisory>
   <Package name="openshift4/ose-metering-hive">
      openshift4/ose-metering-hive:v4.6.0-202112140546.p0.g8b9da97.assembly.stream
   </Package>
</AffectedRelease>
…
<AffectedRelease cpe="cpe:/a:redhat:openshift:4.7::el8">
   <ProductName>Red Hat OpenShift Container Platform 4.7</ProductName>
   <ReleaseDate>2021-12-16T00:00:00</ReleaseDate>
   <Advisory type="RHSA"
      url="https://access.redhat.com/errata/RHSA-2021:5107">RHSA-2021:5107
   </Advisory>
   <Package name="openshift4/ose-metering-hive">
      openshift4/ose-metering-hive:v4.7.0-202112140553.p0.g091bb99.assembly.stream
   </Package>
</AffectedRelease>

Let’s say you were using a version later than the first version, but earlier than the second, like “v4.6.0-202201311100.p0.gf139e12.assembly.stream”. If you matched this against the second version (starting with v4.7) you would get a false positive result. That’s because the 4.6 tag is before the 4.7 one when compared with RPM version semantics.

How Red Hat resolves this problem with RPM security data is with Common Product Enumeration (CPE). Continuing the previous example of scanning the installed container image “openshift4/ose-metering-hive:v4.6.0-202201311100.p0.gf139e12.assembly.stream”, a check of the Red Hat security data reveals that the fixed-in versions map to distinct CPEs:

<AffectedRelease cpe="cpe:/a:redhat:openshift:4.6::el8">
   <ProductName>Red Hat OpenShift Container Platform 4.6</ProductName>
   <ReleaseDate>2021-12-16T00:00:00</ReleaseDate>
   <Advisory type="RHSA"
      url="https://access.redhat.com/errata/RHSA-2021:5106">RHSA-2021:5106
   </Advisory>
   <Package name="openshift4/ose-metering-hive">
      openshift4/ose-metering-hive:v4.6.0-202112140546.p0.g8b9da97.assembly.stream
   </Package>
</AffectedRelease>
…
<AffectedRelease cpe="cpe:/a:redhat:openshift:4.7::el8">
   <ProductName>Red Hat OpenShift Container Platform 4.7</ProductName>
   <ReleaseDate>2021-12-16T00:00:00</ReleaseDate>
   <Advisory type="RHSA"
      url="https://access.redhat.com/errata/RHSA-2021:5107">RHSA-2021:5107
   </Advisory>
   <Package name="openshift4/ose-metering-hive">
      openshift4/ose-metering-hive:v4.7.0-202112140553.p0.g091bb99.assembly.stream
   </Package>
</AffectedRelease>

If we’re able to determine that our installed version was included in the “cpe:/a:redhat:openshift:4.6::el8” CPE product stream, we will only match against its fixed version, and not against the 4.7 product stream. The problem is that there is currently no way to determine the CPE of a particular container image.

Clair works around this problem by parsing the security data into ranges mapped by minor versions. So for the CVE-2021-44228 vulnerability, and “openshift4/ose-metering-hive” container we end up with three vulnerability entries:

• 4.6.x range, fixed in “v4.6.0-202112140546.p0.g8b9da97.assembly.stream”

• 4.7.x range, fixed in “v4.7.0-202112140553.p0.g091bb99.assembly.stream”

• 4.8.x range, fixed in “v4.8.0-202112132154.p0.g57dd03a.assembly.stream”

Don’t we already have scanning in Clair?

This new feature is complementary to the existing Red Hat RPM scanning performed by Clair, because that scanning only matches against RPM security data. Many container images shipped by Red Hat not only include RPMs, but also other non-RPM content, such as Go binaries, Python and Javascript files. The Red Hat Product Security team tracks vulnerabilities in this content and publishes security advisories against the container images which are affected in the Red Hat security data. Without this feature, all of that security data would be ignored by the existing RPM scanner in Clair.

Conclusion

Container image scanning has been added to Clair. In this article we have shown you how you can use Clair to scan container images for vulnerabilities in non-RPM content. The feature works by looking at the Dockerfile contents included in Red Hat container images, parsing the version into a minor version range, and then comparing it against fixed container versions in the improved Red Hat security data.

Learn more

• Red Hat Security Blog

• Red Hat Product Security Center

• Red Hat Product Security Overview