Red Hat product security risk report

If 2021 and 2022 were the years of supply chain security, 2023 was the year artificial intelligence (AI) captivated the world. Throughout the technical press releases, boardrooms, and online discussions, it was difficult not to hear about the advances in generative AI (gen AI) and Large Language Models (LLMs). While much of it was hype and some of it blind panic, the advances and ongoing effects have been real.

While AI presents new challenges and exciting opportunities, Red Hat® Product Security is actively monitoring and engaging in this growing field. Red Hat is strong in data transparency, security-focused development processes within open hybrid cloud platforms, and commitment to empowering customers with risk-based security decision-making tools, positioning us to excel in these types of workloads.

This edition of the risk report takes another comprehensive look at Red Hat’s core security areas. We have included key highlights and statistics from the vulnerability management program, insightful analysis of 2 critical major incidents, and updates to Red Hat’s strengthened secure development program. Additionally, we explore the changing landscape of software supply chains over the last year. To close, we offer a final message from Red Hat Vice President Vincent Danen.

I am confident that the information provided in this risk report, along with Red Hat’s continued dedication to transparency, serves our customers and the larger community well and with valuable insight. Enjoy.

- Garth Mollett, Product Security Lead Architect, Red Hat

Red Hat Security Advisory (RHSA)

Red Hat Security Advisories continue to be the primary notification mechanism for releasing a software update containing a fix for known vulnerabilities in the Common Vulnerability and Exposure (CVE) catalog, since their inception more than 20 years ago. While the format, delivery mechanism, and other details have developed over the years, the primary purpose remains the same.

The upward trend in the volume of Red Hat Security Advisories did not slow down in 2023, as predicted. Instead, the increase from 2022 to 2023 is inline with what we have seen over the past few years. The fluctuations are mostly accounted for by the size and complexity of the portfolio, more than any interesting security trend, because the total reported vulnerabilities actually went down.

Table 1. Red Hat Security Advisory by severity

Common Vulnerabilities and Exposures (CVE)

CVE numbers remain somewhat steady, seeing a slightly lower number in 2023 than we did in 2022, and still considerably lower than peak in 2020.

Vulnerabilities with a Critical severity rating decreased to 12, while we noticed a sharp uptick in vulnerabilities with an Important severity rating. This likely explains the increased number of Red Hat Security Advisories, as Important CVEs are fixed rapidly. Vulnerabilities with a Moderate severity rating, of which we saw a 39 point downturn, are more likely to be bundled with larger planned releases.

As is almost always the case, Moderate severity CVEs remain the most common with reports of Moderate flaws once again making up more than all other categories combined. 

Table 2. Overview of security flaws by severity rating

Tracking of exploitation in the wild presents little surprise with the uptick in Important flaws reflected in an uptick in exploitation.

While it is nice to see these numbers remaining low, it is important to understand that much of this kind of information, from real world attacks, is kept private and confidential and not shared publicly or with vendors. Many attacks also go undetected. When they are detected, the kind of detailed analysis and forensic required to provide this kind of insight are out of reach for many smaller entities. The significant difference between Low and Moderate CVEs compared to Important and Critical CVEs, in part reflects the reality of the exploitability and usefulness in real world attacks; the true numbers are probably higher and remain an opportunity for more research.

Response times

Response time, the speed in which a fix can be released, continues to be a focus for both Red Hat Product Security and many of our customers. While exploitation of Zero Day vulnerabilities that are unknown to vendors or the public remains a real threat from the most well-funded threat actors, the biggest threat from vulnerabilities lies in the time between the details of a flaw being publicly disclosed and when patches are applied.

In order for patches to be applied, they must first be developed, tested and released. In cases where Red Hat is part of the coordinated disclosure process, this begins before a vulnerability is made public. In all cases, significant testing is done by Red Hat prior to releases, in order to ensure confidence and minimize testing cycles required by customers.

For the sake of transparency, Red Hat will compare these numbers point in time to the numbers published in last year's report, which on the surface may appear a little alarming, with a slight increase in average response time for all but Low severity vulnerabilities. In reality, these averages are easily skewed by a handful of harder or slower to fix flaws due to the low total counts, and while we absolutely strive to keep these numbers trending down, some minor fluctuations are not unexpected, especially when point in time snapshots are taken, rather than day to day tracking.

Table 3. Risk exposure time period comparison by severity

Per product breakdown

The 3 major versions of Red Hat Enterprise Linux, 7 through 9, continue to make up the vast majority of Important and Critical security errata. This statistic is expected, given the size and complexity of the distribution and the heavy presence of C code. However, the difference between Red Hat Enterprise Linux® and Red Hat OpenShift® is narrowing, with the latter overtaking Critical errata, releasing almost double the amount of Red Hat Enterprise Linux.

It is worth noting that the CVE count is the number of CVE flaws fixed in a given erratum, which may not all be of Critical or Important impact. This is especially evident in the Middleware numbers, where bundling fixes into planned releases or fixing flaws by rebasing a component, to be based on a new upstream version, means pulling in multiple fixes rather than backporting a single fix to an existing version, which is more common.

Table 4. Important and Critical RHSA CVE count by product

Common Weakness Enumeration (CWE)

To those whose active work is understanding software vulnerabilities, a more interesting statistic is not so much the severity or counts of flaws across components. Rather, it is the type of flaws observed and which ones are more frequently exploited in the wild.

For this information, we look to CWE. While CWEs allow us to classify vulnerabilities, it should be noted that it remains a fairly basic categorization. It does little to consider the complexities of modern exploit development pipelines, where the attacker combines exploitation primitives to achieve desired capabilities for threat actors. It is an interesting data point for understanding where defensive efforts can hit the hardest, even if it only provides part of the picture. 

Uncontrolled resource consumption is in the top spot this year, along with flaws seen in the “Rapid Reset” attacks (more on this later), which are seeing renewed interest. Unsurprisingly, the remainder of the top 5 are all classic memory corruption issues once again, owing heavily to Red Hat Enterprise Linux and code written in the C language.

If we consider a broader time frame, since Red Hat’s Product Security Incident Response team (PSIRT) first started collecting CWE data in 2012, but focus specifically on vulnerability classes with at least 5 separate cases of bugs with known exploitation in the wild, we see a slightly different picture. However, CWE-416 (use-after-free), still takes a front-row seat, coming 2nd only to CWE-20 (improper input validation), which is a much broader and more general class of flaw than the very specific CWE-416. 

The Red Hat Product Security major incident process considers numerous factors to assess if a vulnerability or event qualifies. The first and foremost is risk exposure. This information includes perceived risk to our customers and the Red Hat brand. Security risk from trivial mass exploitation of mission-critical software ranks the highest. However, other kinds of risk, such as the risk posed by misinformation or panic from confusing or over-exaggerated media coverage are also considered.

The purpose of the major incident process is not only to prioritize fixes, as that happens irrespective of major incident status when an issue receives a Critical or Important severity rating, but also to ensure clear, calm, accurate, and well-defined internal and customer-facing communication channels stay open. On the customer-facing side, this is done by issuing additional artifacts such as a Red Hat Security Bulletin (RHSB), Red Hat Insights detection rules, Ansible remediation Playbooks, the usual security errata, and CVE page entries.

This section provides details on 2 of the 3 major incidents when Red Hat Security Bulletins were released in 2023. For each event, Red Hat provided the following detailed information:

• The basics
• The details: for those interested in understanding the inner workings of software vulnerabilities
• The statistics: such as the number of affected product versions
• An estimate of the time it took Red Hat associates to address the issue
• Report on whether the vulnerability was exploited in the wild and the exact time we received reports of exploitation, if available

RHSB-2023-002: Quarkus Security Policy Bypass - Quarkus CVE-2023-4853

The basics: This was a fairly trivial and more easily understood vulnerability. Red Hat build of Quarkus implemented path-based access control in a way that did not fully normalize pathnames in the same way that HTTP request routing was resolved. This led to path-based access control that could be more easily bypassed by inserting additional forward slash (“/”) characters into a pathname when requesting a protected resource.

The details: Several components within the Red Hat build of Quarkus allowed access control policy to be defined on paths. The affected components were QUARKUS-VERTEX-HTTP, QUARKUS-UNDERTOW, QUARKUS-CSRF-REACTIVE, and QUARKUS-KEYCLOAK-AUTHORIZATION. All 4 components contained slightly different implementations of the same pattern mentioned above, where the logic for comparing a match did not properly normalize the paths before comparison, allowing it to be bypassed with path names that would not be exact matches for the policy definition but would be normalized to access the targeted protected resource.

The statistics

Affected major product versions: Red Hat OpenShift Serverless 1, Red Hat build of Apache Camel, Red Hat build of Quarkus, Red Hat Integration 2, Red Hat Process Automation Manager 7
Estimated associate time: 1024 hours
Severity rating: Important
Embargo time: 0 days
Time from public to first fix release: 7 days / 1009 days (security report / original)
Time from public to all fixes released: 28 days / 1092 days (security report / original)
Exploit code published: NA. No code required to exploit.
Exploitation in the wild: Not reported but likely happening on some scale.

Closing thoughts: Two things stand out about this vulnerability. First, it serves as a great example of how complexity makes security a challenging task. What would seem like a trivial task on the surface of the problem, comparing pathnames becomes fraught with difficulty due to the complex nature of the platform and various layers at play. Second, the significant amount of time between public disclosure and then fixed is noteworthy. This delay in the upstream project occurred due to not understanding the security implications in its entirety when the flaw was first reported publicly upstream via a GitHub issue. Some discussions were held on whether this was an expected behavior or a documentation issue. When the issue was reported again nearly 3 years later, the security implication was noted and the fixes were prioritized. 

RHSB-2023-003: HTTP/2 Rapid Reset CVE-2023-44487 and CVE-2023-39325

The basics: Around mid-late August, Amazon Web Services (AWS), Cloudflare, and Google all noticed a massive spike in what looked to be HTTP denial of service (DoS) traffic.

Upon closer inspection of the traffic and attack, it was discovered that the DoS was using a novel attack method and abusing features of the HTTP/2 protocol itself to cause significantly more impact on resource consumption than the traffic would have on its own.

The details: At the heart of the HTTP/2 protocol is the concept of establishing multiple bi-directional communication streams over a single TCP connection. Communication over these streams is multiplexed to allow many requests to happen simultaneously over a single connection without incurring the overhead of establishing multiple TCP sessions or having to allow each request sequentially to complete in serial as is required for earlier HTTP protocol versions.

For this to work, both sides must maintain state machines for each stream, similar to what happens at the TCP layer, whereby the protocol designates how the valid transitions between states occur in order to establish and shutdown streams, as well as transmit and receive data, and handle error conditions. This is all defined in the HTTP/2 RFC, RFC9113, which we’ll summarize.

In order for clients and servers to play nice with each other, the protocol provides several tunable parameters that can be exchanged that allows clients and servers to agree on which settings work best for the session. One of these settings is to advertise the maximum number of streams they will allow the other side to initiate concurrently (SETTINGS_MAX_CONCURRENT_STREAMS). Should either endpoint attempt to initiate a new stream when this count has been reached, the receiving end must reject the attempt with one of 2 possible error codes , PROTOCOL_ERROR or REFUSED_STREAM, delivered in a RST_STREAM frame.

There is an interesting interaction between the state machine, the RST_STREAM frame, and the MAX_CONCURRENT_STREAMS parameter. The state machine has 5 states, which are, IDLE, RESERVED, OPEN, HALF-CLOSED, and CLOSED. Only streams in the OPEN and HALF-CLOSED states count towards the MAX_CONCURRENT_STREAMS count. A stream in any state that receives a RST_STREAM frame, transitions to the CLOSED state, and thus, no longer counts towards the MAX_CONCURRENT_STREAMS count.

Once this mechanism is understood, the attack itself is straightforward. A client endpoint opens a new stream, requesting a resource and then immediately sends a RST_STREAM frame. While this might seem fine from the state-machine perspective, remember that the state machine does not exist in a vacuum and is there to facilitate the communication transport. The resource request may already be enqueued or is being processed, either on the end-point itself or in many cases by another system, when the RST_STREAM frame is received and the stream transitions to CLOSED. This allows the client to open a new stream. If this is done frequently, it will use a significant amount of resources regardless of the MAX_CONCURRENT_STREAMS setting. This issue occurred in August, when the attack took place, allowing a relatively small botnet to cause significantly more impact than was seen in the past.

The statistics:

Affected major product versions: As this was a protocol flaw, almost all products were affected.
Estimated associate time: 6000+ hours
Severity rating: Important
Embargo time: 1 day
Time from public to first fix release: 7 days
Time from public to all fixes released: 22 days
Exploit code published: Yes
Exploitation in the wild: Yes, this issue was discovered due to an active attack in the wild and was added to the CISA Known Exploited Vulnerabilities (KEV) catalog within the month.

Closing thoughts: It is unusual for a flaw that has no impact on the confidentiality or integrity of a system to be a major incident, but that is what we saw with Rapid Reset. Since this was a protocol flaw, a great deal of software was affected, making the response efforts extra costly, as seen in the massive estimate of associate hours. We also saw this flaw discovered because of an active attack against 3 very large hyperscalers, which had to exert considerable effort to defend against the attack, even with the mammoth scale at which they operate daily.

A special thank you is in order to AWS, Cloudflare, and the Google Cloud Platform. They did not just work to block the attack for their customers but provided considerable technical analysis that allowed the open source community to work on getting fixes into various code bases impacted by the flaw to reduce its usefulness and value as a future attack vector. 

This year, we continued to increase our focus on secure development by standardizing more of our Secure Development Lifecycle (SDL) practices. Our steadfast focus on the SDL process prioritizes one key objective: mitigating risks for our customers. We go beyond simply aligning with industry standards like the NIST Secure Software Development Framework (SSDF). Red Hat has a goal to deliver every product and service as inherently trustworthy and reliable. This commitment enhances the value of our portfolio and strengthens customer trust in our brand.

Red Hat uses the open development model to foster innovation resulting in technologies that are more trustworthy and stable. We actively contribute to hardening and testing open source software upstream, providing a stronger foundation for all customers. Downstream, we rigorously apply industry-leading secure development frameworks like the SSDF, exceeding the common security requirements of NIST SP 800-53. We achieve this by aligning our stringent internal security standards with NIST SP 800-218 (SSDF) and leveraging the controls outlined in NIST SP 800-53.

Our comprehensive security controls include Static Application Security Testing (SAST), Dynamic Application Security Testing (DAST), Threat Modeling with architecture and code reviews, and Penetration Testing with automated scanning and white box assessments.

The following sections will take a more comprehensive examination into these activities and showcase key security achievements through statistics and highlights.

Threat modeling

Red Hat conducted a total of 105 threat model engagements in 2023 that uncovered 310 implementation weaknesses. While none of these weaknesses met the bug-bar to count as vulnerabilities in and of themselves, some of them likely would have led to vulnerabilities in the future, if the hardening opportunities had not been addressed.

Red Hat rates these hardening opportunities on a scale similar to the Low, Moderate, Important, Critical (LMIC) scale used to rate vulnerabilities, but with an additional level below Low known as Informational. The table below shows the total counts for each level.

Figure 5. Implementation weaknesses findings

Penetration testing

The Red Hat Product Security internal Penetration Testing (PenTest) team conducted a total of 39 PenTest engagements in 2023, finding 101 implementation weaknesses and 17 exploitable vulnerabilities, ranging from informational to full remote code execution issues.

Figure 6. Penetration testing findings

Rapid DAST (RapiDAST)

RapiDAST is a security testing tool, initially developed for internal use by the Product Security PenTest team in 2021, however, now it has been released as an open source project available to the community through GitHub.

RapiDAST provides an automation and reporting harness that allows various scanning tools to be configured and run against live application API endpoints. Providing report generation in both human and machine readable formats and integration with reporting solutions such as Open Worldwide Application Security Project (OWASP) DefectDojo.

While the tool is offered with no support, contributions are welcome and we will continue development of the tool ourselves as we integrate it further into Red Hat’s delivery pipelines.

Red Hat Product Security took an even more proactive approach to enhancing the security of our software supply chain. In 2023, Red Hat laid a strong foundation for our risk-based software supply chain security approach by enforcing stringent security standards throughout our software supply chain.

Every system handling code undergoes a rigorous security operating approval process before integration into production pipelines. We identified pipeline systems and defined processes to close security gaps, leading to a 20% reduction in technical debt through decommissioning non-essential assets. This significantly reduces security risks associated with unpatched and unmanaged systems.

In 2023, we proactively addressed potential supply chain risks (listed below) through our comprehensive security practices, upholding our commitment to risk-based product security.

Threat landscape

In 2023, the Linux Foundation surveyed 691 organizations globally on their current use of open source technologies, resulting in these top 3 responses: Linux (64%), cloud and containers (63%), and Continuous Integration and Continuous Delivery/Deployment (CI/CD) and DevOps tools (54%). Naturally, this heavy reliance on open source technology makes it an attractive target for software supply chain attacks (SSCA).

Software supply chain threats confronting development organizations continue to rise, and open source is no exception.

Based on our analysis of 52 publicly reported SSCA events in 2023, Red Hat assessed that 79% of SSCAs had unknown threat actors (see Figure 7 for known actors) and 67% lacked evidence of motive for the attack. The following represent the 2 most typical attack scenarios:

• Lower probability, high-impact scenario: Attacks conducted by a well-resourced and sophisticated threat actor that profoundly affects an organization’s security posture, resulting in code tampering with broad-reaching implications for upstream and downstream software supply chains.
  • These attacks represented approximately 17% of publicly reported SSCAs in 2023.
  • The most notorious example is the SolarWinds attack in 2020.
  • The most recent example is the 3CX Desktop App attack in March 2023.
• Higher probability, lower impact: Opportunistic attacks by threat actors of varying maturity targeting victims with easy-to-exploit weaknesses or vulnerabilities in the open source ecosystem with an increasingly decentralized software supply chain. Whether an attack is indiscriminate or targeted, both cybercrime groups and state-sponsored threat actors abuse the same weak links in software supply chains and use similar attack vectors. However, their resourcefulness and outcomes differ.
  • These attacks represent approximately 83% of publicly reported SSCAs in 2023.
  • Adversaries usually aim for compromised tokens to authenticate access to infrastructure and services hosted on AWS, Google platforms, or GitHub.
  • This scenario can be a prequel to a lower probability, high-impact SSCA.
  • Threat actors use common techniques such as typosquatting, repojacking, dependency confusion, or compromised user accounts to gain initial access. 

Based on our trend analysis of SSCAs from 2022 to 2023, software developer secrets remain the highest attraction and most coveted by threat actors. What is known as the “Secret Leak” technique represents the technique most often used on both sides of the cyber kill chain. For example:

• Open Software Supply Chain Attack Reference (OSC&R): Under the “Impact” tactic or phase of the kill chain, the Secret Leak is the ultimate goal for a threat actor, as demonstrated in the high-profile breach of the CI/CD vendor CircleCI in January 2023, resulting in the leak of assorted API secrets.
• MITRE ATT&CK: The same Secret Leak event could be the stepping stone that advances a threat actor to the Initial Access phase, where techniques are applied to compromise a software supply chain. Bypassing the Reconnaissance and Resource Development phases of the cyber kill chain saves the adversary time and resources.

According to our data analysis, 81% of the SSCAs in 2023 were connected to abusing popular open source platforms such as node package manager (npm) and Python Package Index (PyPI), with a notable increase in the number of malicious packages detected on these platforms. Publicly available data shows a significant shift in the malicious targeting of PyPI more than npm. 

Red Hat prioritizes risk management for our customers. This year showed a fairly significant uptick in Important severity vulnerabilities, both discovered and exploited, and this trend will likely continue. This is where attention to what matters becomes crucial. We must focus on addressing the immediate risks that, if exploited, pose the greatest risk to customers and their deployments.

Again, we see that this pragmatic approach pays dividends as the majority of exploited software continues to be in Critical and Important severity issues. Add to that our ability to pivot more quickly and address those vulnerabilities that are seeing active exploitation without available fixes, and our focus on addressing real risk is apparent. Promptly applying patches will protect customers from negative outcomes associated with software exploitation.

Using data such as Common Weakness Enumeration (CWE) to target classes of vulnerabilities is of great interest. Approaches like using memory-safe languages are intriguing, but bring other problems. As we saw, the type of weakness with the most vulnerabilities this year was one where memory-safety would not have been beneficial (CWE-400), yet for the rest it would have. Authoring code in memory safe languages is a good approach to take when writing new code, but is less attractive when we consider well-written and long-established code that would need to be rewritten. Anytime we change code, and especially when it's new, opportunities abound to introduce new, unknown vulnerabilities. An influx of new software could cause vulnerability counts to rise further, at least in the short term.

Finally, it is certainly disconcerting to see the continued attacks on open source supply chains. Red Hat is largely insulated from these attacks through our curation process of software that we select, as well as using internal repositories to store the code we use. Through our focus on stability and minimal code changes, we do not go back to upstream often enough to fall afoul of watering hole attacks and similar. In this way, obtaining core open source from a vendor such as Red Hat is an effective way to minimize the effects of these supply chain attacks. However, users must recognize that not all open source will come from a vendor and they must continue to be diligent, ensuring processes are in place to help protect against attacks on software pulled directly from upstream projects and repositories.

Using Red Hat software provides protection against these sorts of attacks, but effective protection relies on a shared responsibility. Customers must be aware of what is consumed from upstream and, in either case, be able to apply patches more quickly to protect their assets.

- Vincent Danen, VP Product Security, Red Hat

• For more information about Red Hat vulnerability management processes, you are invited to read An open approach to vulnerability management.
• For the latest information on security for Red Hat products and services, visit Red Hat Product Security center.