An article from December 2020 reported that 2020 had a record high number of CVEs reported for the fourth year in a row (yet another reason to dislike the year!). Across the technical spectrum more than 17,447 CVEs were reported. Back when we started the Red Hat Risk Report the volume of CVEs across all software vendors numbered in the 4,000-8,000 range. The specific reasons for the increase will be debated for some time to come, but the harsh reality is that the organizations need to address a growing number of vulnerabilities each year.
In the 2020 Red Hat Product Security Risk Report we can clearly see this trend of increased reports and increased patches:
Out of this ocean of data and threats that have severely impacted many companies large and small, how can you know where to take action and where you may be able to delay your response to address issues of higher concern?
One school of thought (one I’ll spend the balance of this post attempting to counter) is to market or brand the flaw to raise attention to it. Let’s dive into a brief history of marketing security vulnerabilities and see if we can discover any actual benefits from this trend, perhaps even finally answering the universal question:
Does branding a security vulnerability help you fix it any faster?
Mr. Peabody, his boy Sherman and their Wayback Machine can help us here. Let’s jump in and set the dial way back to 2014. CVE-2014-0160 erupted onto the scene in April of that year. Researchers had discovered a very severe flaw in OpenSSL which was (and remains still) a very important component not only in Linux, but a multitude of other operating systems, software platforms, and devices. No one can say precisely if Heartbleed was the very first "celebrity" or "named" or "branded" security vulnerability, but the lineage of current trend of branding is an arrow that leads straight back to my bleeding heart (branded flaw pun?). Naming these things most likely came out of the historic practice of naming malware - worms and trojans, which was more often than not based on the name of the actual file being executed or called (Blaster, SQLSlammer, etc). Modern naming has evolved a bit.
At the time, the state of vulnerability management was much less mature than it is today in understanding, tooling, and process. So the researchers decided to "brand" the flaw with a logo and a website, still found here as Heartbleed Bug.
Terrifying isn’t it? To a lay-person or corporate executive it certainly captivates both the eye and the imagination. Given the wide-spread severe impact of the flaw, this might have been merited at that time. Heartbleed got mostly everyone’s attention. It was featured on 24-hour cable news networks, and even my mother asked me about that "bleedy heart thing." Thanks Mom for knowing what I do!
Shellshock escalates the hype
Then, what started out as a joke on Twitter led to the next major branded event: Shellshock. Articles like "Shellshock is bigger than Heartbleed says experts" started to escalate the branding arms race. Each new finding had to be "bigger" and "scarier" than the one that came before it.
In 2015 we wrote our first major public thoughts on this emerging trend: Don’t judge risk by the logo. In this insightful post, Red Hat Product Security reviewed a string of more severe issues and their consequences all without hype, any PR/Marketing, or an Emcee. This became a core theme of our daily operations and our core mission to provide clear, calm advice on issues that matter.
Time progressed, more vulnerabilities were discovered and fixed, several groups tried to impress the community with their cleverness in naming and art. The trend became so troublesome that in 2016 a researcher came prepared for disclosure with a full-on marketing campaign for their issue, Badlock, only to have the internet respond with a collective "meh."
Backlash to the campaign was quick. With all due respect to the researchers, yes there was a problem that needed to be fixed, but coming out of the gate as harbinger of the apocalypse for something that is much less severe certainly reduces credibility and calls attention away from legitimately bad problems.
In 2017 we created our Customer Security Awareness (CSAw) program with the goal of cutting through the over-hyped noise and providing deep details and actionable steps for those impacted by these high-profile security flaws. This program seeks to collect technical details, detection tools, and provide dispassionate and objective explanations of these large, industry-impacting vulnerabilities that impact our portfolio and our customers.
Meltdown and Spectre: Do names really help?
Now jump to 2018 to "that melty ghost thing" as my mom called it when she spoke to me on January 4th. The Industry was faced with a problem that impacted virtually every device that had a CPU. The problem was found through some amazing research by several teams and as the industry grappled with how to fix it the immense scope and complexity of the issue rapidly became apparent. These related attacks were the first in a string of research that continues to this day, but the boggle for security responders was trying to articulate what it all meant.
Very hard to execute and complex to fix, Spectre (CVE-2017-5715 and CVE-2017-5753) and Meltdown (CVE-201-5754)-captivated the press and CSIRT responders like few other vulnerabilities before or since (outside of the granddaddy Heartbleed). While it was easy to shorthand to "Spectre and Meltdown," that disguised the nuanced complexity of the underlying flaws.
Did having the names help? Aside from easing conversations and getting someone introduced to the core concepts quickly (which was vital for the "Spectre-like" and other following vulnerabilities that were discovered and corrected later), we propose that the name itself did not help get an issue of this scale and scope fixed more effectively than those that simple bore a CVE-moniker.
We’re not alone in this view of the suspicion on the value of the marketing and branding of these flaws. Friend-of-the-show Art Manion, CERT/CC, and the Carnegie Mellon Software Engineering Institute very vocally joined the fray with their Arbitrary Albatross presentation and subsequent Vulnonym: Stop the Naming Madness post. The naming and hype can lead to either a waste of time/effort/resources responding to this never-ending stream of silly named things to the detriment of real issues, or the general public could become so numb from all the noise and cutesy logos that they ignore the truly critical problems.
Heartbleed five years later: Where are we?
Which brings me to the 2019 article "Five years later, Heartbleed vulnerability still unpatched." In 2017 Shodan noted that over 200,000 vulnerable devices still lurked on the internet according to their scans. Two years later Heartbleed continued to linger on despite all the press, hype, and patching.
Celebrity vulnerabilities are a crutch for immature or incomplete vulnerability management programs. Companies, no matter what the size, industry vertical, or geography need to have reasonable processes in place to be informed about vulnerabilities that affect the technologies they deploy.
The economics of vulnerability disclosure
Let’s speak a bit to the economics of vulnerability disclosure, which will help shed some light onto the motivations for using these PR-style tactics. When looking at researchers (called Finders within our FIRST PSIRT community) there is no one label that applies across the board to everyone doing this work (and it totally is hard, generally thankless work, that Red Hat Product Security and our peers very much appreciate and attempt to ensure that proper credit is given for that work).
Researchers can be just that: someone investigating something. They may be a student or academic seeking a degree or furthering the common body of knowledge in some area. These researchers also could be professional bug hunters, someone that devotes their time and makes their income by finding and reporting and getting paid for security vulnerabilities.
A Finder also could be an open source community developer, maintainer, or downstream supplier of that component that sees some odd results during testing or gets a bug report from their downstream about strange behavior. A Finder also could be an altruist, seeking to make the world (and the code) a better place as they find a problem.
Sadly, Finders also can be "bad guys and girls" (which they sometimes are), leading to attacks, "zero days", and general nonsense. Let's agree we want all the other Finders to discover these defects before that group!
As we stated: there are a lot of security vulnerabilities floating around out there across all products and services. As an admin/operator, it certainly is challenging maintaining and patching a fleet of heterogeneous systems and applications while still keeping up to date on new findings and updating your ecosystem appropriately.
Imagine making your living on reporting these same vulnerabilities or that this thing is the basis of your research for your thesis or dissertation. Whether it is for monetary compensation, "fame," or to further your research or academic career, there is a competition for the attention of the community, conferences, or the press for this "fortune and glory" around being the smartest person on the internet to find the "very important thing." Researchers have leaned into the celebrity vulnerability mindset to attempt to pull their way to the top of the stack, get noticed and get paid, so to speak.
Reducing fear, uncertainty and doubt (FUD)
To bring us full-circle back to our 2020 Risk Report, a quick survey highlights many branded vulnerabilities throughout 2020. Of those, only two rose to the level that Red Hat Product Security felt additional materials and explanation on the matters were merited. Trying to reduce the fear, uncertainty, and doubt in vulnerability disclosure has been one of our core objectives over the years.
Reflections on 2020 Vulnerabilities
What can be said about 2020 that hasn’t been said already? It definitely was a year where things happened and there certainly were several of those things that involved security. Looking across the vulnerability landscape, we see that more than 176,447 CVEs were reported.
Within the Red Hat portfolio, we identified 2,040 unique CVEs that impacted components we supply and support. This was far-and-away the highest volume of CVEs we’ve fixed in any calendar year on record. This translates to a significant amount of work an operator or administrator needs to do in order to help see to it their systems are running at peak patch levels.
We understand that most enterprises do not run exclusively on Red Hat products and services and as someone that is responsible for a heterogeneous environment that has a melange of technologies to keep updated it can seem like a Herculean task.
This is one of the reasons for our Red Hat Severity Scores that we issue with every single vulnerability along with our CVSS scoring and CWE analysis. Every security issue has some level of importance to deal with, but some issues have higher likelihoods of being exploited or have higher consequences if they were.
It is interesting to note that over the years we’ve actively tracked and reported on issues impacting our software to see the change in distribution of the severity of issues. The volume of Critical and Important issues that we consistently address across the whole portfolio have remained generally flat, with a slight uptick in 2020, but are nowhere near "record levels." Red Hat Engineering addressed Critical issues across the portfolio with great speed. 31% of CVEs that we rated as Critical were addressed and had patches for consumers within one business day. A total of 89% had fixes within one week and a full 100% were addressed within one month of public disclosure.
Overall the volume of issues we patched was 1.5 times higher than we had in 2019 with the average and median delivery times being down, which translates to faster availability of security updates. The volume of Moderate security flaws that were fixed in 2020 alone was more than ALL the vulnerabilities Red Hat fixed back in 2011, 2012, 2013, and 2014 (plus we fixed 460 Low severity issues as icing on that cake). This was a 3x increase in volume across the board since 2011...what do the next nine years hold? Only time will tell.
As systems get more complex, the key to reducing your risks associated with them is to have effective patch and vulnerability management programs in effect and to minimize the attack surface you present a malicious or curious actor.
It is worth noting that when default security features are disabled (like turning SELinux off for example, which if you did would make Dan Walsh cry), the risk profile of that system is drastically altered, opening up the potential for additional security risks and impacts. Good security hygiene, timely patch management, and appropriate access controls and logging can go a very long way preventing the next terrible media headline from impacting you.
We hope you’ve enjoyed this series of blogs around our 2020 Product Security Risk Report. Each of these articles has expanded upon a concept covered within the report, so if you liked the blogs, please read the full report to learn more.