In part 1 of this story we traced the history of Red Hat Product Security from its inception in 2001 through to its shift into the Customer Experience and Engagement (CEE) team in 2013.
But that was just the beginning...security was always important, of course, but it was about to become front-page news.
The rise of branded vulnerabilities: Heartbleed and Shellshock (2014)
2014 saw the rise of “branded” vulnerabilities —security issues that were given memorable names and logos, and that were accompanied by big splashy announcements.
The purpose of this was reportedly to increase public awareness of these issues, with the hope that increased attention would result in faster fixes.
Of course, increased awareness also led to increased media coverage, which, in turn, led to more public awareness. While these branded security issues were serious, the widespread attention tended to result in somewhat overblown panic and drama.
The Heartbleed vulnerability was disclosed to the OpenSSL team on April 1, 2014. It was a serious vulnerability in the OpenSSL cryptographic software library which allowed information to be stolen that would normally be encrypted.
The severity of this bug was largely due to the quantity of internet communication that was secured by OpenSSL. This included much email, web, instant messaging (IM), and virtual private networks (VPNs). Heartbleed allowed attackers to steal data, eavesdrop on communications, and impersonate services and users.
Heartbleed was patched six days later, on April 7, 2014. Even though Heartbleed's impact on Red Hat products was minimal, it had a great deal of impact on the industry and shaped the evolution of Red Hat's response to such security events.
Shellshock is a family of security bugs in the Unix Bash shell. Bash was created as part of the GNU Project as a free replacement for the Bourne shell, and is often installed as a system’s default command line interface (CLI).
Shellshock (originally called Bashdoor) was initially disclosed to Bash’s maintainer on September 12, 2014. The bug allowed an attacker to execute arbitrary commands on vulnerable systems by bypassing restrictions to the environment. Certain services and applications allow remote unauthenticated attackers to provide environment variables, allowing them to exploit this issue. A number of related vulnerabilities were discovered shortly afterwards.
Shellshock was patched and announced publicly 12 days later, on September 24, 2014.
Unfortunately, attackers were quick to take advantage. Within an hour of the vulnerability being announced, there were reports of machines being compromised due to the bug. And the number of attacks quickly grew as attackers deployed botnets while administrators struggled to update impacted systems.
Red Hat’s response
Garth Mollett, Product Security Lead Architect, explained how Red Hat reacted to these new branded vulnerabilities.
“The amount of attention these were getting meant that we needed to refine some of our processes. We couldn’t treat a big event like this the same as we treated every other vulnerability that came in. Heartbleed was front-page news, and all of our corporate communications people were just getting smashed with requests about it.”
“We ended up developing what we called a ‘high-touch process’—later renamed to CSAw, which was short for customer security awareness, and now simply referred to as the Red Hat Product Security Incident Response Plan. This was a cross-team, cross-department process that allowed us to actually have a communications plan in place, and to make sure that every part of the company that was going to be inundated with questions was ready to go. We’d have a press release ready in some cases, and have articles ready to be published. The point was to make sure that all the information was available, and that everyone was reading from the same book.”
The team started putting this process into place in response to Heartbleed, and when Shellshock came along, it was able to respond more quickly and more decisively to help lead the industry response.
The Product Security team not only helped guide the communication and messaging about Shellshock across the industry, but also helped create and coordinate the patches.
“We actually wrote a bunch of the patches there, and were helping to coordinate the disclosure across all the major Linux vendors. And it ran surprisingly smoothly. None of these things go 100% to plan, of course, but it was very rewarding. I had someone come up to me at a conference and thank me for the work that Red Hat did, which was a strange experience.”
The Spectre Meltdown era (2018)
The Spectre and Meltdown vulnerabilities were publicly disclosed in January 2018, however the Red Hat security and engineering teams had known about them for quite some time, and had been working on fixes under embargo with Intel and industry peers.
Unfortunately, in early 2018, both Spectre and Meltdown were leaked, forcing the team to release planned fixes out to customers as quickly as possible.
What are Spectre and Meltdown?
Spectre and Meltdown were different but related hardware flaws that allowed attackers to gain unauthorized access to system data.
Spectre allowed a local attacker to leverage a flaw in how CPUs use branch target injection and how the CPU performs bounds-checks to bypass security controls and expose potentially sensitive data.
Meltdown allowed a local attacker to leverage a flaw in how CPUs use speculative execution to read data in system memory to expose potentially sensitive data.
The scope of these flaws was enormous—affecting Intel, AMD and ARM architectures—so they quickly generated a huge amount of interest among customers, the public, the media and government.
Red Hat’s response
Huzaifa Sidhpurwala, Principal Product Security Engineer, was part of the Product Security team at that time.
“We worked on this flaw for a very long time. But what happened on January 3, 2018, was that there was an early leak of the flaw. This was a big effort because it not only involved Product Security, but various people from the Engineering team, Quality Engineering, Marketing, Products and Technologies, DevOps, Legal, Information Risk and Security, and a lot of other teams were involved.”
Sidhpurwala continued, “It’s one of the most expensive flaws we’ve worked on so far. In the end it involved multiple rounds of advisories for the issue, improving the mitigations once public, and more than six months of effort in engineering in actually writing the code and doing quality tests etc.”
Of course any early leak is going to cause a bit of panic since it’s unexpected and forces security teams to quickly prioritize their work. “If we are working on 20 products,” said Sidhpurwala, “which product is the one that has the greatest customer base? What errata goes second? And so on. Trying to figure out where most of our customers are is what is most important in this kind of situation.”
Clifford Perry, Senior Manager, Product Security, was also part of Red Hat’s response. “We are under constant fear of an embargo breaking like it did on January 3. Thankfully we had most of our response ready to go. Even though we lost some time, our response (videos, explainers, collaboration, trustworthiness, etc.) was appreciated by customers and partners alike, and increased our credibility within the industry.”
Expanding into managed services (2019)
In 2019, Red Hat started offering a lot of its new and existing products to customers as managed services, where Red Hat provided hosting for customers, rather than giving them software to install and use in their own datacenters.
Vincent Danen, Red Hat’s Vice President of Product Security, explained, “This was the first real increased scope for Product Security. As more of our products were moving to the cloud, we had to address the security aspects of that. We had to figure out how to treat services the same way in terms of vulnerability response and some of the more proactive security things we wanted to do.”
The result was a new managed services team within Product Security. This new team would do things like penetration testing, threat modeling, and some more proactive security work on those services. “We knew the threat surface of managed services was greater, and the risk to Red Hat was higher as well since we were hosting customer workloads and data.”
And it wasn’t a solo effort. “We were working with a number of other teams, like Information Risk and Security, the OpenShift Dedicated team, the OpenShift SRE teams, and compliance. Managed services was a unique way for Product Security to engage with a lot of other teams within Red Hat to come together and collaboratively support those services.”
SolarWinds hack and protecting the open source supply chain (2021)
In February 2021, the SolarWinds breach and other significant security events highlighted weaknesses in software supply chain security.
Unlike Heartbleed, Spectre and the other branded vulnerabilities, “SolarWinds” is the name of the company whose systems were breached. SolarWinds was also different in that it wasn’t really the result of a software bug, but the result of SolarWinds’ own systems being compromised.
An attacker was able to acquire superuser access to SolarWinds’ process of verifying the integrity and authentication that gave attackers trusted and highly privileged access to a vast range of other networks, including a number of prominent U.S. and U.K. government organizations.
The SolarWinds breach very quickly made software supply chain security a top priority concern for the entire software industry.
Red Hat’s response
The SolarWinds breach did not impact Red Hat's products, services, customer data or assets but the event brought more attention to open source supply chain security.
And while Red Hat has always worked to ensure the safety and security of its open source supply chain, the team focused on these efforts expanded in both size and scope after the SolarWinds event.
The future of Product Security at Red Hat
When asked what the future looked like, Danen responded:
“When thinking about the future of Product Security, there’s so much possibility and it really depends on what Red Hat needs, what our customers need, and where we can make a positive impact on the industry when it comes to open source security in general. These three areas are our primary focus and we adapt as the business and the industry adapts.”
Danen mentioned an increased focus on the ‘secure life cycle management’ and curation of open source code, and about creating a resilient and trusted supply chain for bringing in open source code and preparing it for enterprise customers.
“This means working with upstream communities, and our developers who are a part of them, to write code with security in mind, which benefits everyone. It means ensuring our productization pipelines—those systems that make that upstream code suitable for enterprise support—are protected and trustworthy.”
Red Hat’s Product Security team has a long and storied history of being open source leaders, influencers and contributors, and it intends to continue on that path well into the future.
Danen said, “In the wider industry, we have opportunities to influence what the next generation of security metadata, automation, and vulnerability management looks like.”
“The challenge isn’t just to make our software better and more trustworthy, but to enable the customers who use our software as a platform to host or run their own, to also create trustworthy code for their customers. We do this by influencing an ecosystem of software development, be it in our software or their consumption of code from upstream open source communities. This really is the next frontier—enabling and empowering upstream and downstream open source communities and users to use open source securely.”