What is backporting, and how does it apply to RHEL and other Red Hat products?

Version numbers are important, but aren't always what they seem at first glance. Red Hat, for example, often backports updates to the software we ship in Red Hat Enterprise Linux (RHEL) to maintain the version that we shipped.

This is a post to follow to Jean-Sébastien Tougne’s post on finding the latest available kernel. Jean-Sébastien’s article was responding to a question on the Red Hat Learning Community, where the poster was seeking the latest version of the kernel for Red Hat Enterprise Linux. That prompted me to write an article that went deeper into the nuance and strategy the Red Hat Enterprise Linux team employs for this to be magically delicious for administrators.

Why are versions important?

The first question is: why are versions important?  In some cases that is very straightforward, newer software means new features and new capabilities. For example, RHEL 8 updated established subsystems like systemd and added new features like Application Streams, Image Builder, and more.

Sometimes what comes with a new version is less dramatic, like when we released RHEL 8.1. The new stuff from RHEL 8 was still there, but we added a System Role for managing local storage and newer versions of software packages delivered through the Application Streams mechanism like NGINX 1.16.

Then we get down to the most tedious and detailed answer to the question of why versions are important? Software has bugs. They range in severity, scope, and exploitability. Users request new features or changes to packages. Closing of these bugs and requests is also reflected in updated versions of packages that are provided as part of the Red Hat Enterprise Linux distribution.

Where would one go to find all this information? There is not a single answer to that question in terms of what is new in a major release. Red Hat’s product marketing teams do a lot of work getting this information out there. Be it through updates to redhat.com product pages, blogs, talks at conferences like Red Hat Summit or other industry shows, and through materials for customer-facing employees and partners. Likewise, the next level of detail, the Engineering and documentation teams at Red Hat produce release notes like this one for Red Hat Enterprise Linux 8.1.That final level of detail, bugs, vulnerabilities, etc. is what the rest of this article is going to address.

Understanding versions in RHEL

Before we dive into the details of what’s been applied to this or that version of software distributed with Red Hat Enterprise Linux, let us first discuss how the software distributed with Red Hat Enterprise Linux is maintained. We refer to some of this process as backporting, but what is it and how does it happen?

If you look at an RPM package distributed with Red Hat Enterprise Linux, you will notice a consistent naming convention. As an example, take a look at the Bash shell package. To get this information, you could run yum list bash or, in the example below, I used 

rpm -q bash

Here's how it breaks down: 

Name: Name of the software. In this case, the Bash shell.

Version: The version of the open source project used to make this package. In this example, this package is based on version 4.2.46 from the GNU Project, the maintainers of Bash.

Extra vers[ion]: This portion of the version on this package is Red Hat specific. Generally, RPM provides this field for the packager of the software to maintain their own version specific data. Because this package comes from Red Hat, this extra version data is being applied by the Red Hat maintainer of this RPM. Different maintainers or teams at Red Hat may choose to format this field slightly differently, but it has some versioning information and usually indicates the version of Red Hat Enterprise Linux that the package was made for.

Arch[itechure]: The CPU type the binaries contained in this package are compiled for. You could see other CPU architectures in this field if the package binaries were compiled for a different CPU (e.g. aarch64), or noarch, which would indicate that the package is architecture neutral.

How backporting works

Now we are digging into more details, what did Red Hat do to make 31.el7 and how that is different than earlier versions of the bash-4.2.46 RPM package? What we are talking about now is backporting. Literally, we are taking code from a newer version upstream and integrating it back to an older version of the software. This is one of the great things about open source.

When Red Hat Enterprise Linux 7, the version running on the example machine I am using, was branched from Fedora upstream, Red Hat engineers pulled the source for Bash 4.2.46 from Fedora's package. The Red Hat engineers continue to maintain this copy of source code distributed with Red Hat Enterprise Linux 7. 

As changes are made upstream to fix bugs or address vulnerabilities, Red Hat's engineers may take the changes from upstream and integrate them to update the version shipped with RHEL 7. Not all changes are backported, of course. We are strictly looking to address bugs and security issues, rather than trying to bring new features to older releases.

What does this look like to the people administering a machine? If you take a look at the changelog of a package, in our case bash, using rpm -q --changelog bash you would see something like the following, note that this log is formatted newest change to oldest, so you may want to pipe the output to less to see the newest changes:

From this output, we can see the history of the package as well as what was changed in each of the versions of the RPM that were issued. The most recent change is the one that generated version 4.2.46-31.

In addition to the description of what this update resolves, readers are also given the Red Hat bugzilla ID number (#1495398) that corresponds to this, now resolved, problem. Alternatively, down in the update that resulted in version 4.2.46-28, a security vulnerability was addressed (CVE-2016-9401). If the short resolution details here in the changelog are insufficient for what you need to know about this update, you can access these additional resources to find more in-depth information.

Why would Red Hat choose to operate RHEL packaging this way? Why go through all the effort of backporting instead of just downloading updated source from the open source project and compiling a new version?

Red Hat Enterprise Linux is a complex distribution with a lot of different packages that make up the product. Further, on top of this platform many organizations run their own workloads and software (which has their own complexities). If Red Hat replaced Bash with a completely new version, would this change affect any of those other packages or workloads running on top of the distribution? 

New releases of Bash may have removed or changed the way some features (like, say, POSIX-related behavior) that could affect scripts or software running on top of RHEL. That's fine for the upstream release or future versions of RHEL when users are expecting it, but bad for a release that customers expect to be stable between updates.

By using this method of backporting, Red Hat is able to provide updates to the operating system, but reduce risk historically associated with applying updates to complex operations infrastructure.

Putting it all together in a real-life scenario

A number of organizations have security requirements that mandate system updates be applied within a certain number of days or months. In order to measure the success of compliance with these policies, systems can be audited with security scanners. 

However, most scanning software we've encountered does not account for the backporting of features that Red Hat uses for maintaining and updating packages in the Red Hat Enterprise Linux distribution. As an example, connecting to the SSH daemon running on my Red Hat Enterprise Linux 7 system shows a version of:

openssh-7.4p1-21.el7.x86_64

However, the most recent version of OpenSSH from openssh.com is 8.1, not the 7.4 being reported from the system.

The likely outcome would be me getting a notification from the auditor telling me that I need to update the ssh package on my system. However, this is likely not needed since I have the latest, supported package from Red Hat installed on the machine. While my machine is reporting openssh 7.4, it’s also showing an extra version of 21.el7, and if needed, I can pull data from the RPM changelog to check that vulnerabilities that the auditor may be concerned about have been closed through backported fixes.

I hope you can use this knowledge of how Red Hat maintains Red Hat Enterprise Linux software through the practice of backporting, along with the sources of information about releases and command-line tools to get even more depth about specific changes in your career and organization.