Issue #22 August 2006

The Fedora Project and Red Hat Enterprise Linux
Part 4 of 4: Red Hat Enterprise Linux development

By Tim Burke

Welcome back to the final installment of this four-part mini-series explaining and contrasting the development and usage models for Fedora Core™ and Red Hat® Enterprise Linux®.

So far, we've reviewed the goals and objectives of the Fedora Project and of Red Hat Enterprise Linux. Then we took a good look at the participants and the types of development that goes into each distribution. In the last article we focused on the collaborative development of Fedora Core.

This final article will detail the approach used for the development of the Red Hat Enterprise Linux distribution. It concludes with a summary of why this dual development model is a win-win for all involved as a summary of all four installments.

How to get involved with Red Hat Enterprise Linux

There are two primary means to effect change in Red Hat Enterprise Linux. They are direct and indirect:

Indirect Red Hat Enterprise Linux involvement
Much of the content in Red Hat Enterprise Linux is directly inherited from its respective upstream development tree. For example, Apache development comes from the Apache organization, and kernel development comes from Linus' kernel.org trees.

Consequently, if you wish to get a major new kernel feature incorporated into Red Hat Enterprise Linux, the place to start is on kernel.org. From there the feature will be inherited first into Fedora. Provided that the feature proves to be successful in Fedora, it will later make its way into Red Hat Enterprise Linux when it inherits from the respective Fedora Core baseline. In this manner, Red Hat issues a formal invitation not only to our key customers and partners, but also the broader open source community to participate in this revolutionary new approach to operating system development.

Direct Red Hat Enterprise Linux involvement
In the interests of not destabilizing the established customers for a given release, we typically do not incorporate new features during the maintenance cycle of a release. For example, for customers running Red Hat Enterprise Linux 4, major new features are likely to be deferred to Red Hat Enterprise Linux 5.

The primary work in Red Hat Enterprise Linux 4 is security enhancements, bug fixes, performance enhancements, hardware enablement, and minor new feature additions in response to strong customer request. The main means of interacting with Red Hat for bugfixes is through our support services. There are established partner management channels with Red Hat's major hardware and software partners. Through the development of Red Hat Enterprise Linux maintenance releases we often do co-development and testing activities with these strategic partners. This greatly increases the capacity of work that can be collectively accomplished, and it also broadens the testing matrix.

The typical development model for Red Hat Enterprise Linux is that the packages get rebased (replaced with new major versions) for each major release (e.g. Red Hat Enterprise Linux 3, Red Hat Enterprise Linux 4). We use the experience and feedback of Fedora Core releases to evaluate which version has been demonstrated to be product-ready. Once a version of a package has been selected for a Red Hat Enterprise Linux release, such as Red Hat Enterprise Linux 4, we typically stay with that version of software and apply incremental fixes to that version. This approach is taken to minimize the change rate and to maintain stability.

In contrast, the typical way that problems are resolved in Fedora is to just try the latest-and-greatest version of the package. Ongoing fixes in Fedora typically consist of wholesale package rebasing, whereas Red Hat Enterprise Linux fixes typically consist of identifying small changes to address just the issue in question.

In the course of developing the next major Red Hat Enterprise Linux release (i.e. the successor to Red Hat Enterprise Linux 4) we work closely with both community developers and major business partners on co-development and testing activities. For example, the vast majority of kernel enhancements targeted for inclusion in Red Hat Enterprise Linux 5 are being directly developed upstream in kernel.org. Similarly, the compiler enhancements slated for Red Hat Enterprise Linux 5 are being developed in the gcc community. These enhancements are first inherited and tested in Fedora and later in Red Hat Enterprise Linux.

For this reason, we also encourage our co-development partners to work their new feature enhancements into the respective upstream pool. This way we all benefit from the community model.

Red Hat Enterprise Linux Release Vehicles

The release schedule for Red Hat Enterprise Linux consists of new major releases and update releases. New major release are designated with numbers, such as Red Hat Enterprise Linux 3, while update releases are designated with U, such as Red Hat Enterprise Linux 3U7. These update releases occur at intervals of four to six months.

Requirements for updates are aggregated and prioritized by the Support and Partner Management teams. For each update there is a beta version, which customers and partners have access to. Given the fact that Red Hat Enterprise Linux updates are focused on stability, all changes incorporated must meet stringent selection criteria.

For example, changes must:

  • Address a demonstrated issue encountered by customers. They must be more than just theoretical or corner-cases.
  • Preserve compatibility. Changes are not allowed to break ABI/API interfaces. This is even applicable to kernel-related changes where preservation of the kernel binary interfaces (e.g. kABI) is required.
  • Be only minor feature enhancements. We do not incorporate invasive and/or risky new features into a Red Hat Enterprise Linux update.

High-profile bug and security fixes are asynchronously released as-needed. These new versions of Red Hat Enterprise Linux packages are referred to as "errata." The analogous term for Fedora is "update."

Red Hat Enterprise Linux major releases occur at intervals of approximately 18-24 months. Most Red Hat Enterprise Linux releases have one or more initial test versions, each referred to as an Alpha baselevel. The term "Alpha" marks early test versions prior to feature freeze.

Following the Alpha are usually one to three Beta baselevels, such as "Beta 1" and "Beta 2." Beta 1 is the first feature-complete baselevel. Beta 2 includes bugfixes to issues encountered during Beta 1.

After the Betas comes the Release Candidate (RC) baselevel. If no egregious issues are found with the RC version it will be blessed as the final General Availability (GA) version and become the officially shipping release. Red Hat Enterprise Linux customers and partners are allowed to access the alpha and beta test releases.

For more information on the Red Hat Enterprise Linux development model, refer to the following articles:
Linuxjournal.com: Constructing Red Hat Enterprise Linux 4
Linuxjournal.com: Constructing Red Hat Enterprise Linux v. 3

Conclusion

Red Hat openly interacts with individuals, customers and partners to deliver robust open source distributions, as shown above. We recognize that there are different, and often conflicting objectives of the respective audience. The tailored approach has proven to be much more effective than a "one size fits all" strategy. Better for everyone involved.

No login required. Want to see your comments in print? Send a letter to the editor.

This is why Red Hat hosts the Fedora Core community distribution to foster rapid development and integration of open source projects. Red Hat further hardens, tests, and provides ongoing support in the form of Red Hat Enterprise Linux for business users. There are well-defined means of participating in Fedora as a volunteer developer and more indirect means of effecting change on the Red Hat Enterprise Linux release.

In the end, this "best of both worlds" approach yields a win-win for all involved. The power of community development yields rapid innovation and broad testing. The discipline of structured Red Hat Enterprise Linux releases yields a stable platform for business-critical deployments. Each of the approaches feed back to their respective upstream development team to share in both the bugfixing and feature enhancement. This unwavering commitment to open source community development has consistently been the hallmark of Red Hat's operating model and has proven to be even more sound today as at the company's inception.