Assessing Red Hat Ansible Automation Platform vulnerabilities

What your security scanner isn’t telling you

Security, more than ever, needs to move with speed, and we hear much about “shifting security left” and DevSecOps as methods to help achieve this. As this new paradigm gains momentum, so does the reliance on automated security tools to identify and mitigate software vulnerabilities at scale.

But what if these security tools aren’t telling you the full story?

Often, our customers reach out to us saying their security scanners flag Red Hat Ansible Automation Platform as insecure, or that it contains unpatched vulnerabilities. Rest assured, our products are security-hardened and battle-tested. Red Hat's long-standing track record of upstream contributions extends to improving upstream projects' security and contributing to industry standards. The real culprit here is your security scanner!

In this blog, we’ll cover:

• Red Hat’s approach to product security, including Ansible Automation Platform.
• Why security scanners generate false positives for Red Hat products.
• Available resources you can use to make informed product security decisions.

Note

Several links in this blog point you to resources in the Red Hat Customer Portal, which requires a user account. You and members of your team can register online or reach out to your local Red Hat team to get started if you need to create an account.

Red Hat’s approach to product security

Red Hat looks at product security at the portfolio level

When considering Ansible Automation Platform product security, it’s important to note that Red Hat manages its ecosystem as a portfolio. This approach means product security, service-level agreements (SLAs), and assurances included in the Red Hat Subscription apply to all our products, including Ansible Automation Platform.

Ansible Automation Platform retains the security posture of Red Hat Enterprise Linux

Ansible Automation Platform and RHEL.

Over 90% of Fortune 500 companies trust and use Red Hat products, and Red Hat Enterprise Linux (RHEL) is renowned for its stability and security. Ansible Automation Platform utilises the same battle-tested, hardened RHEL RPM Package Manager (RPM) packages and productization standards you trust.

For example, Ansible Automation Platform 2.2.1 installs the Subversion RPM during the setup process. Below is a snippet of the dnf info subversion-libs command output from a Google Compute Engine RHEL 8 instance with Ansible Automation Platform installed. 

# dnf info subversion-libs
…output omitted…
Installed Packages
Name         : subversion-libs
Version      : 1.10.2
Release      : 5.module+el8.6.0+15157+188c9801
Architecture : x86_64
Size         : 5.1 M
Source       : subversion-1.10.2-5.module+el8.6.0+15157+188c9801.src.rpm
Repository   : @System
From repo    : rhui-rhel-8-for-x86_64-appstream-rhui-rpms
…output omitted…

Terminal output - dnf info subversion-libs.

Note that Ansible Automation Platform used the official Red Hat @System repository for the subversion-libs package. 

Does this include automation execution environment RPMs? Yes!

Automation execution environments use the same security-hardened RPMs and RHEL Universal Base Image (UBI) utilised throughout our product portfolio.

Automation execution environments.

Let’s verify this by opening a Bash shell in the Ansible Automation Platform supported execution environment using the following ansible-navigator command:

# ansible-navigator exec /bin/bash --eei registry.redhat.io/ansible-automation-platform-22/ee-supported-rhel8:1.0.0-162

ansible-navigator exec command.

Once the shell opens and the RPM package is installed, we can run the same command used in the previous RHEL 8 instance example.

bash-4.4# dnf info subversion-libs
…output omitted…
Name         : subversion-libs
Version      : 1.10.2
Release      : 5.module+el8.6.0+15157+188c9801
Architecture : x86_64
Size         : 4.3 M
Source       : subversion-1.10.2-5.module+el8.6.0+15157+188c9801.src.rpm
Repository   : @System
…output omitted…

Supported EE dnf info subversion-libs command output

Note that the Subversion source RPM version used in the supported execution environment is identical to the version used in the previous RHEL 8 instance command output:

subversion-1.10.2-5.module+el8.6.0+15157+188c9801.src.rpm

Accelerated release cycles using execution environments

Execution environments accelerate our engineering team’s capability to deliver Ansible content and software package updates.

Red Hat Ecosystem Catalog supported EE dashboard.

The example above is the Red Hat Ecosystem Catalog entry for the Ansible Automation Platform supported execution environment.

Supported execution environments are available in the Red Hat Ecosystem Catalog and published with insightful information, such as the release date and the Container Health Index. We’ll cover the Container Health Index in more depth later in this blog.

Next, we’ll go through Red Hat Product Security's approach to identifying and mitigating vulnerabilities.

Identifying and reporting vulnerabilities

Despite the rigorous tests, efficient development models and the latest and greatest development tools, “Bug” happens - in proprietary and open source software offerings. When these bugs have security implications, they’re called vulnerabilities.

Red Hat Product Security

Red Hat vulnerability management workflow.

Red Hat Product Security, a dedicated team of software security specialists, reviews and analyses every vulnerability that’s reported to Red Hat.

Red Hat Product Security uses its in-depth knowledge of our product portfolio and security-focused supply chain practices to determine if the vulnerability impacts a Red Hat component. If further action is required, Red Hat Product Security generates security information and resources using industry-standard formats.

CVE-2022-24070 details a vulnerability introduced in the subversion-libs module -  part of Subversion version 1.10, which made it vulnerable to memory corruption. We’ll use this CVE as an example to illustrate the information Red Hat Product Security generates.

CVE-2022-24070 - Red Hat Customer Portal dashboard

Common Vulnerabilities and Exposures (CVE) identifier

Every fixed CVE has a public entry in the Red Hat CVE database, available on the Red Hat Customer Portal.

Common Vulnerability Scoring System (CVSS) score

CVSS scores measure the impact of a vulnerability, not its risk. Red Hat uses the CVSS base score as part of a more holistic impact assessment.

Note

Open source software CVSS scores may vary based on multiple factors. We recommend using a CVSS score provided by Red Hat for our products.

Red Hat Severity Rating

Along with the CVSS base scores, Red Hat rates the severity of a security issue using a four-point scale; Low, Moderate, High, and Critical. Red Hat Severity Ratings help you make informed decisions to plan and prioritize upgrade tasks.

Common Weakness Enumeration (CWE) classification

CWE is a community-developed list of common software and hardware weakness types. Every vulnerability impacting Red Hat products will be evaluated using CWE for root cause analysis.

Mitigating vulnerabilities

Finding a flaw is only the first step. Bugs need evaluation and, if necessary, remediation using a mature product life-cycle process. Red Hat provides full life-cycle support as part of the Red Hat Subscription and releases patches based on the support life-cycle phase and Red Hat severity rating.

Red Hat uses two techniques to deliver software updates; backporting and rebasing. These methods enable our products, derived from multiple upstream projects and communities, to continue providing enterprise-grade stability.

Rebasing

Rebasing is replacing an existing package with the newer upstream version. When a package is rebased, Red Hat updates the full version number to reflect the latest upstream version, not just the Red Hat release number.

Backporting

Backporting is when a software feature, enhancement, or fix is taken from a more recent version and applied to older versions of the same software. Red Hat uses this technique to support backward compatibility with dependent components and to reduce the risk of introducing new bugs.

Let’s continue with the CVE-2022-24070 example to illustrate backporting.

CVE-2022-24070 - Subversion backporting example

The above screenshot shows that Red Hat patched subversion version 1.14 and backported the fix to version 1.10 under the Red Hat Security Advisory (RHSA)  RHSA-2022:2234.

A closer look at the RHSA highlighted below confirms that Red Hat patched subversion-libs version 1.10.2 to mitigate CVE-2022-24070.

 RHSA-2022:2234 - subversion-libs patched version.

In other words, upgrading to a later version of subversion-libs to mitigate the vulnerability is unnecessary. Red Hat backported the fix to version 1.10.2.

And this is where your security scanner isn’t telling you the full story!

Security scanners - Reading between the lines

Accurately identifying vulnerabilities is a complex task; typically, security scanners only perform a simple version check to determine vulnerabilities. This approach can lead to false positives and, perhaps even worse, false negatives.

Beyond the version number

Red Hat increments product release numbers when backporting fixes to identify the newer, patched RPMs, commonly referred to as the N-V-R (Name-Version-Release) software versioning strategy.

Let’s continue with the subversion-libs example to illustrate this.

# dnf info subversion-libs
...output omitted...
Installed Packages
Name         : subversion-libs
Version      : 1.10.2
Release      : 5.module+el8.6.0+15157+188c9801
Architecture : x86_64
Size         : 5.1 M
Source       : subversion-1.10.2-5.module+el8.6.0+15157+188c9801.src.rpm
Repository   : @System
From repo    : rhui-rhel-8-for-x86_64-appstream-rhui-rpms
...output omitted...

Terminal output - dnf info subversion-libs.

The command output above shows the subversion-libs package details installed on the RHEL 8 instance. The package version number is 1.10.2, and the release number is 5.module+el8.6.0+15157+188c9801. This version matches the patched version released in RHSA-2022:2234.

The diagram below illustrates the Subversion RPM N-V-R.

Patched subversion N-V-R.

When the toast hits the floor, scanner-side-down

You guessed it: If your security scanner only uses version numbers to determine RPM vulnerabilities and ignores the release numbers, it will incorrectly flag subversion-libs version 1.10.2, creating a false positive for Ansible Automation Platform. Quick searches of third-party vulnerability databases confirm this.

In one example, CVE-2022-24070 is marked as High Severity, and the advised remediation is to upgrade to subversion-libs version 1.14.

In another example, the vulnerability database didn’t mention release numbers. Instead, it only specified that subversion 1.10.0 through version 1.14.1 in their entirety are vulnerable! 

Both examples imply upgrading Subversion to mitigate the vulnerability. Thanks to backporting, upgrading is unnecessary and reduces the potential of introducing new bugs, incompatibilities, and, worst of all, new vulnerabilities.

What security scanner should I use?

As we saw in the examples above, third-party security vendors rarely incorporate the common Open Source Backporting practices to determine patch levels and system integrity.

The Red Hat Vulnerability Scanner Certification program, offered in collaboration with third-party security scanner vendors, helps you receive precise results when scanning Red Hat container images and packages.

Certified vulnerability scanners utilise real-time Red Hat security data to determine if a product is affected by a vulnerability and gain deeper insight into the patching status.

Certified security scanners are available in the Red Hat Ecosystem Catalog.

Let’s look at additional resources available to help you accurately assess the Red Hat portfolio for vulnerabilities.

Security resources - Red Hat has you covered

Red Hat is committed to providing the security tools and data you need to identify and mitigate vulnerabilities across your Red Hat estate. We release vulnerability information affecting our portfolio in the Red Hat Customer Portal using industry-standard formats and make it easy to use with tightly-integrated Red Hat management tools. 

Below is a list of available tools and data sources to help secure your Red Hat environment.

Red Hat Insights for Ansible Automation Platform

Red Hat Insights for Ansible Automation Platform, available on Red Hat Hybrid Cloud Console, provides solutions to proactively resolve infrastructure performance issues, system availability, and security vulnerabilities using the latest security information, our technical experience, and industry best practices.

Red Hat Insights for Ansible Automation Platform remediations.

The Insights user interface example above indicates an Ansible Automation Platform RHEL 8 host is vulnerable to CVE-2018-12126. Insights for Ansible Automation Platform provides the option to remediate the vulnerability using an auto-generated Ansible Playbook.

Red Hat Ecosystem Catalog Container Health Index

Ansible Automation Platform publishes supported execution environments in the Red Hat Container Catalog. It provides a grading system similar to the Red Hat Severity Rating, called the Container Health Index (CHI).

Calculating the Container Health Index grade

The CHI helps you identify and mitigate security risks in container image RPMs published in the Red Hat Ecosystem Catalog. Red Hat scans container image RPMs using internal and public advisory and vulnerability resources to determine the image grade. As new advisories become publicly available, we review the Red Hat Ecosystem Catalog inventory to determine if any image RPMs are affected and update the CHI grade accordingly.

Let’s continue with the Ansible Automation Platform supported execution environment example to explore the CHI in the Red Hat Container Catalog.

Supported EE Container Health Index.

The above example shows the supported execution environment Health Index (A) and a reference to security advisory RHBA-2022:7244.

RHBA-2022:7244 details a vulnerability patched in version 2.2.5-8.el8_6.3.x86_64 of the expat package as part of the RHSA-2022:6878 security errata.

RHSA-2022:6878 - expat patched version.

Running  dnf info expat in the ee-supported-rhel8:1.0.0-162 execution environment confirms the updated expat version is installed, mitigating the vulnerability.

bash-4.4# dnf info expat
Red Hat Universal Base Image 8 (RPMs) - BaseOS                                                                                                           3.2 MB/s | 813 kB     00:00
Red Hat Universal Base Image 8 (RPMs) - AppStream                                                                                                        9.7 MB/s | 3.0 MB     00:00
Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder                                                                                                 78 kB/s |  20 kB     00:00
Installed Packages
Name         : expat
Version      : 2.2.5
Release      : 8.el8_6.3

dnf info expat command output.

Note

The CHI rating calculation includes unapplied, yet available, security fixes to the underlying products and does not account for vulnerabilities present where a patch is currently unavailable.

Security data

Red Hat provides security data in multiple industry-standard formats, which include:

• Open Vulnerability and Assessment Language (OVAL) definitions.
• Common Vulnerability Reporting Framework (CVRF) documents.
• Common Security Advisory Framework (CSAF) files.
• Vulnerability statements and acknowledgements.
• Vulnerability metrics data.

You can use these data sources to generate metrics unique to your environment, keeping us honest and accountable.

RPM and container image signing

Red Hat signs all our RPM-based packages and container image content using the Red Hat signing server. This capability verifies that you’re using original packages from trusted sources.

Red Hat Security Data API

The Red Hat Security Data API exposes a list of endpoints to query Red Hat security sources to retrieve CVRF, CVE, and OVAL data. It also supports JSON, XML, and HTML output formats.

Red Hat Incident Response Plan

The Red Hat Incident Response Plan details the recommended mitigation process with clear, concise advice to address arising security flaws.

Red Hat Product Security Risk Report

Red Hat publishes an annual report that contextualises the vulnerabilities and threats that impacted our product portfolio over the past calendar year (2021 report). 

Red Hat RHSA mailing list

Anyone interested in receiving Red Hat Security Advisories (RHSA) can subscribe to the rhsa-announce@redhat.com mailing list.

In summary

The Red Hat portfolio - the RHEL security footprint is Ansible Automation Platform’s 

Red Hat manages our ecosystem as a portfolio, and the same tried and trusted packages and container images used throughout our ecosystem comprise Ansible Automation Platform. We strive to continually improve our overall security perspective to protect customers, contributors, and partners from digital security threats.

Your security scanner isn’t telling you the whole story

Security scanners typically try to solve the complex issue of vulnerability identification with a simple approach - checking package version numbers. This method is insufficient and can lead to false positives and, perhaps more harmful to your security, false negatives.

Security scanners you can trust

Red Hat provides the Red Hat Vulnerability Scanner Certification program in collaboration with security scanners to get the accurate data you need when scanning Red Hat products.

A unified set of data sources and solutions

We offer a unified solution to vulnerability identification, mitigation, and management that includes data sources, tools, and platforms made available as part of the Red Hat Subscription to help secure your Red Hat environment.

Resources and where to go next

• An Open Approach to Vulnerability Management - an excellent document going into further details discussed in this blog. I highly recommend reading it.
• Red Hat Product Security Blog - the latest news and technical knowledge to help you secure your environment.
• Red Hat Ansible Security Automation - Learn more about how we can help you with security response and remediation in your environment.
• Self-paced lab exercises - We have interactive, in-browser exercises to help you get started with Ansible Automation Platform.
• Free trial subscription - Try Ansible Automation Platform in your environment.
• Red Hat Ansible Automation Platform YouTube channel.

A special thanks to Red Hat Product Security and the authors of An Open Approach to Vulnerability Management!