20 Years of Red Hat Product Security: From inception to customer experience (Part 1)

From its inception in 2001, the Product Security team has been focused on providing Red Hat’s customers value in the form of hardened and streamlined security updates and testing across the entire product line, including managed services and, most recently, our own open source software supply chain.

But more than that, they’ve also been essential members of the wider open source security community, helping ensure the openness and transparency of security vulnerability information, aiding in the fight against open source fear, uncertainty, and doubt (FUD), and being key contributors in the response to a wide array of high profile security incidents and vulnerabilities.

So, let’s take a look at how Product Security has evolved at Red Hat over the past 20 years, and at what the future might look like.

Red Hat Product Security is established (2001)

It’s not often that you can trace the existence of something like a security team back to one specific day, but the Red Hat Product Security team was formed September 21, 2001.

We know this because it all began with a single phone call between two Red Hatters—Paul Cormier, then EVP of Engineering and Research & Development, and Mark Cox, one of the few people then at Red Hat who had a formal background in security.

Cormier had recently joined Red Hat, and, given his involvement with a security company previously, he realized that Red Hat needed a dedicated security team to ensure that every issue and vulnerability was properly and promptly dealt with.

That dedicated security response team was initially just Cox, who worked alone for many years. Now, the Red Hat Product Security team is significantly larger.

Thinking back to those days, Cox said, “Everyone at Red Hat just wanted to do the right thing and they were really enthusiastic about the project, about what we were doing for the industry and about what we were doing for open source. They really believed in that and in those values, which were the same then as they are now. Today it’s the same passionate company it used to be.”

Embracing CVE (2002)

The Common Vulnerabilities and Exposures (CVE) naming system is a community-driven effort to provide a centralized catalog of reported security vulnerabilities that was formally launched in September 1999. 

Every reported vulnerability is given a unique CVE record and ID, which are then used to communicate consistent descriptions of the vulnerabilities so security professionals can be sure that they’re talking about and working on the same issues. As of late 2021, there are more than 174,000 CVE records, each available in multiple human and machine-readable formats.

Red Hat’s involvement with CVE started in 2002. Cox had developed an efficient workflow for handling vulnerabilities within Red Hat, and since the security team felt that CVE made a lot of sense, they decided to go all-in.

“I became a board member,” said Cox. “We were one of the initial organizations that started assigning names for all issues. It gave us a way to track an issue right from the initial port through the final fix. It was really important for Linux distributions to have these common names because we all ship different upstream versions of each of the packages. And sometimes we did backport, at which point it became really important. We had to know for certain that we were talking about the same issues.”

On the flip side, however, there were a number of entrenched entitites that were spreading a lot of FUD (fear, uncertainty and doubt) about Linux and open source security issues, while not reporting security issues they found in their own software. “If they found and fixed a security issue themselves,” Cox explained, “they would not assign it a name. They’d just fix it silently.” The lack of transparency on their part made it difficult for anyone to really understand how the different systems actually compared, security-wise.

Cox, who continues to be a CVE board member to this day, believed that the CVE naming system meshed well with Red Hat values. “We at Red Hat were open. We were transparent. We wanted to be true to our values, and the best way to do that was with CVE.”

Fighting open source FUD (2005)

In 2005, Red Hat decided to step up the fight against open source FUD by making security metadata publicly available on the website, including all of the metrics and mapping between different data sets.

Vincent Danen, Red Hat’s current Vice President of Product Security, explained, “It was a strategic move that was designed to disrupt what some people in the industry were doing about vulnerability counting. They were trying to compare the number of vulnerabilities in, for example, a Windows product versus a Linux product, trying to compare them as if they were similar. Publishing this data allowed Red Hat to show that not all vulnerabilities were created equal, and that we fixed those that mattered quickest.”

In addition to publishing this vulnerability data on the Red Hat Customer Portal, we also publish it via an application programming interface (API) that was made available in 2016, making the data very easy to access. Danen said, “We borrowed the idea a little bit from Cisco, who had something similar, but at that time it was behind a paywall. We were determined to make this freely available for customers or anyone who wanted to make use of it.”

The Red Hat security metrics data includes all CVE metadata, including Open Vulnerability and Assessment Language (OVAL) feeds and Common Vulnerability Reporting Framework (CVRF) feeds, available in XML format for downloading or through the API in JSON format.

Streamlining security updates (2007)

The next significant evolution in Red Hat Product Security came in 2007 when the team deployed hardware signing keys to streamline and improve the security around the process of signing package updates.

While the Product Security team was only eight people at this point, it was being recognized throughout the industry for the quality of its work.

“We had Microsoft saying that we were the best-practice example for security response amongst Linux distributions,” said Cox. “Symantec’s Internet Threat Report called us out as having the best track record of dealing with third-party vulnerabilities.”

Of course, there was always room for improvement.

Prior to 2007, every time a package update came out for Red Hat, one of the few trusted engineers had to log into a special server and use an “extremely long and annoying” passphrase to manually sign packages, which happened almost daily.

The existing signing key had other issues as well. In a post to the Red Hat Security blog in January 2007, Cox wrote:

“[T]he authorised signers not only had the ability to sign with the key, but they also ha[d] the ability to read the key material. In theory this means that a malicious internal signer could copy the key, take it away with them, and sign whatever and whenever they wanted. Or, more likely, a clever intruder who gained access to our internal network could perhaps capture the key and passphrase, compromising the key. The risks mean we've had to be really careful who has signing privileges with the legacy key and how the key signing is handled.”

What’s more, in 2007 there were a growing number of cyberattacks against U.S. government agencies. It was clear that the team needed to do something, and do it quickly. This is when it decided to generate new keys in hardware and distributed those with Red Hat Enterprise Linux (RHEL) 5.

“The obvious solution was to use a hardware signing module,” said Cox, “but it wasn’t like we could just buy one off the shelf to do the job.” Since there was no off-the-shelf solution available for hardware-based RPM key management, the team developed one itself, using a combination of existing security modules and custom patches.

Not only did this new hardware signing module simplify the process of signing packages, it also prevented unprotected key material from being exported. Authorized users would be able to sign with the new key, but never have access to the key material itself.

“Customers wouldn’t have noticed any difference, but we were saving them from possible security events,” said Cox. “By doing this and doing this early, we ensured that customers would have less to worry about in the future.”

Customer experience and engagement (2013)

Red Hat’s Product Security team has always been focused on doing what’s best for customers, of course, but in 2013 an organizational adjustment solidified this both internally and externally. The team shifted to become part of the Customer Experience and Engagement (CEE) team.

Cox explained the reasoning behind the change. “The placement of Product Security is really important to get right, to send the right message to both internal and external customers. And CEE was the right place at the right time. Our focus has always been on doing the right thing for customers, making sure that everything had a CVE identifier, that we weren’t hiding vulnerabilities, and that we were working to fix vulnerabilities quickly. Everything we were doing was about providing customer value, so CEE made a huge amount of sense.”

The move was well-timed for another reason, but the team wasn’t aware of it at the time.

“It turned out this move was timed exactly right because the next year was when we started to get the branded vulnerabilities, where the hype and actual severity of an issue bore no relevance to how important the issue actually was,” explained Cox.

As part of the CEE team, Product Security was better positioned to communicate with customers (and the community as a whole) to help coordinate the industry response to these major headline-grabbing security events.