The Fedora Project and Red Hat Enterprise Linux
Part 3 of 4: Fedora Core development

by Tim Burke

-->

Welcome back to our mini-series contrasting the Fedora™ Project and Red Hat® Enterprise Linux® and describing how we use these 2 distributions to meet the needs of customers and community participants.

So far, we've reviewed the goals and objectives of the Fedora Project and of Red Hat Enterprise Linux. We also examined the participants, highlighting what type of development goes into each distribution.

This article focuses on the collaborative development of Fedora Core.

Fedora taxonomy

Red Hat Enterprise Linux is an operating system distribution produced by Red Hat. The Fedora Project is a broad umbrella organization whose charter is to promote a wide selection of open source projects. These open source projects range from the development of new software to documentation, translation, and volunteer testing.

The Fedora Project helps build online communities with mailing lists and other forums.. The range of Fedora projects is much broader than the selection of packages that ultimately find their way into Red Hat Enterprise Linux.

In order to encourage as much participation as possible, Fedora introduced the concept of "Fedora Extras." "Extras" is a rendezvous point where interested volunteers can add their own open source software.

There are standards in place to ensure that content submitted to Extras meets basic requirements and undergoes an initial quality screening. The wide-open nature of Extras accommodates diverse content. For instance, there may be many different packages in Extras which perform similar functionality. This fosters a "may the best packages win" environment.

Red Hat Enterprise Linux does not provide multiple programs that accomplish the same task. There are many reasons for this approach in enterprise software. It is impractical to expect systems administrators to have to test multiple applications for every new deployment. There are also practical space limitations on CD images, as well as long-term support implications. Fedora extras avoids the space constraint of a CD image by not burning CDs at all; it is entirely electronically distributed.

The Fedora Project packages that are collected and distributed on CD installation media are referred to as "Fedora Core."

What types of software can be included in Fedora?

For Extras, almost anything can be considered--as long as it runs on Linux. A "sponsor" is required. One of the Fedora maintainers reviews the software to see that it would be interesting to at least some of the Fedora community. Additional instructions for packaging the RPM, resolving dependencies, and other basic requirements can be found at fedoraproject.org.

There are also a few things that are explictly forbidden.

The Forbidden list includes rules like:

• If it is proprietary, it cannot be included in Fedora.
• If it is legally encumbered, it cannot be included in Fedora.
• If it violates United States federal law, it cannot be included in Fedora.

The Fedora development model

The Fedora Core distribution is developed by aggregating 1,500 distinct packages (including applications, libraries, documentation, kernel, and management utilities) from the package's respective upstream community. Here's a few examples:

• The Apache web server has its own development community. Red Hat engineers monitor and participate in development within the Apache community. The Red Hat engineer identifies the latest reasonably stable version of Apache and integrates it into Fedora.
• A Red Hat engineer involved in the Samba project selects the right version to include in Fedora. Over the course of one Fedora Core release, a package may be updated numerous times.
• Red Hat engineers actively participate in the development of the Gnome user interface widgets and are interested in exposing these capabilities through Fedora. This gives developers valuable feedback and garners suggestions for improvement and expansion.

If the majority of Fedora packages are inherited from their respective upstream codebase, what development does Red Hat do in Fedora?

• Interaction issues - A lot of the work is ironing out interactions and dependency issues. Often when you update foundation components it has a ripple effect. For instance, a new version of the compiler may have more stringent compile time error checks. This typically results in a range of issues which must be resolved in the affected packages.
• Project management - Red Hat helps establish release schedules, coordinating which major versions of packages best facilitate tracking dependency issues. Release engineering work like constructing build infrastructure, composing ISO CD images, and developing electronic distribution mechanisms also happens at Red Hat. Typically, new Fedora Core releases occur every 6 months; the average span between major releases of Red Hat Enterprise Linux is 18 months.
• Bug fixes - Red Hat developers resolve a large number of integration issues and work with the respective upstream community to feed back the fixes
• Installer and system management utilities - Tools that automatically detect hardware and install appropriate drivers, enable the user to select packages of interest, and tailor the system to site specific needs are also developed at Red Hat.
• New feature development - While much of the community development work is done outside of the Fedora source tree, there are a number of projects lead by Red Hat developers with their primary source code repository under the Fedora umbrella. Examples include Red Hat Global File System (GFS), a clustered filesystem, as well as Directory Server.

How to get involved with Fedora

The communication hub for Fedora is http://fedoraproject.org/. At this site you will find descriptions of the various Fedora projects, and can download pointers and advice on how to get involved. Participation is welcome in many different capacities, such as:

• Ambassadors help spread the word about Fedora, Linux, and open source, and recruit new volunteers.
• Documentation teams produce content to help end users and developers.
• Testing and bug triaging groups act as a bridge between users and developers.
• Developers add enhancements or bugfixes and integrate completely new software.
• Translators work on text translation and input methods with the goal of making Fedora useful to a worldwide audience.

Since the focus of this article is on development, let's dig deeper into how developers become more involved with Fedora. The fedoraproject.org pages detail how to participate in bug fixes. If your intention is to incorporate sizable new features into Fedora, the following section details how this is typically accomplished.

Things to know before attempting to integrate a new feature into Fedora

1. Identify the primary upstream development site for the feature. For example, the primary repository for the Apache web server is outside of Fedora (as Fedora primarily integrates this external package). Some packages, such as the Directory Server are hosted directly in Fedora.
2. Identify whether the feature is appropriate for inclusion in Fedora Core or whether it is better suited for Fedora Extras. The majority of new Fedora content is placed in Extras. This allows maximum flexibility for the developers as development is not affected by the overall Core distribution schedule.

Projects suited for inclusion in Fedora Core tend to be infrastructure components which have dependencies on a large number of commonly used system utilities and services. A goal of the Fedora Core project is to continue to shrink the number of packages included in Core and relocate as many packages as possible to Extras. This is being done for practical reasons. It would allow Core to fit on a reasonable number of CDs and prevent it from consuming colossal amounts of network bandwidth to download and update.

To perform development work on a feature which has a primary upstream development site outside of Fedora, interact with the developers on that site. This way, the main development work for the respective feature is conducted in a unified community. Fedora then will inherit snapshots of this new software at regular intervals.

The majority of new Fedora feature development is not done directly within Fedora. The Fedora Project can then focus on integration work. Fedora has no intention of forking projects that have established, vibrant development teams.

Commonly, people come to us asking for new feature enhancements to Fedora. In most cases, we refer the requester to the respective upstream development crew. Once the new feature has been integrated into the upstream package (i.e. Apache), it will be included in Fedora the next time the package is rebased. "Rebase" is a term which refers to replacing the version of source code for a given package--in other words, exchanging an older version with a new version.

Fedora development is typified by frequent rebasing. The frequency of rebasing varies considerably by package. For example, during most development windows, the kernel may be rebased several times per week. Other packages may only be rebased once every 6 months. This largely depends on the rate of change in the respective upstream development pool.

If you have determined that your new feature is most appropriate for inclusion in Fedora Extras, there are guidelines describing how to get involved.

If you wish to effect change in Fedora Core packages, the first place to start is the respective upstream community of the package you wish to change. If you want to get a new packages incorporated into Fedora Core, the best place to start this discussion is on the fedora-devel-list@redhat.com mailing list. You can subscribe to this list.

What types of features are best suited for Fedora development?

The Fedora Project is the hub of Linux integration and the perfect place to iron out interaction issues among packages and get broad user testing under many types of workloads. Here are some historical examples of the kinds of projects well-suited for Fedora development:

• SELinux is the implementation of fine-grained security capabilities. This feature required modification of a large number of system services. The implementation goal was to make the enhanced security optional without disrupting behavior end users have grown to expect. The ability to quickly turn around fixes and get broad end-user testing made Fedora the ideal development laboratory. The majority of this work was performed by Red Hat engineers in conjunction with government-sponsored engineers.
• Modular X has substantial system impact, ranging from the build system, hardware/kernel interaction, distribution packaging, installer, and package dependencies. This was a major integration effort, benefiting from broad community development and testing in Fedora.
• Kexec/Kdump is a new crash dump capability. This project has several parts, including kernel infrastructure and hardware driver components. There are a set of associated commands and utilities used to configure and yield system crash dumps. Also there is a set of crash analysis tools used to examine dump images. The ability to collect the various components and these capabilities to a many users makes Fedora an ideal hosting environment. Much of this work has been driven by engineers employed by Red Hat's business partners working closely with our core development team.

Features well-suited for Fedora:

• Are comprised of many different components and therefore benefit from the integration exposure.
• Are receptive to quick feedback cycles.
• Benefit from testing exposure on a wide diversity of hardware and workloads.
• Should have broad user appeal.

For example, an esoteric feature which a minority of the Fedora user base has interest in is unlikely to garner much involvement. This is particularly true when trying to get such features into Fedora Core. Recall, the objective over time is to get Core down to a smaller foundation set.

Esoteric packages are better suited for inclusion in Fedora Extras.

Fedora Core release vehicles

New full releases of Fedora Core (i.e. FC4, FC5) appear approximately every six months. The milestones are structured as follows:

• Rawhide - Throughout the entire development cycle the latest-and-greatest versions of software packages are referred to as "rawhide". Rawhide is typically built on a nightly basis and is the perfect venue for participating in bleeding-edge development. Rawhide allows developers and testers to easily to pull the latest releases from the repository, install, and test. This is frequently where integration issues are first detected. Usage of rawhide is not for the faint-of-heart--it is frequently unstable. For this reason, there are interim release milestones where more stable versions are available.
• Test1 & Test2 - These Test releases are the major development baselevels where big and possibly destabilizing features are prepped for integration. This takes place early in the release cycle. Test1 and Test2 have an approximate duration of six weeks.
• Test3 - This is the last development baselevel. Minor features and bugfixes are incorporated here. Incorporating large disruptive features is strongly discouraged.Test3's primary objective is stabilization. The duration of Test3 is about six weeks.
• Final - "Final" is the term given to the last baselevel. Only bugfixes to issues encountered in the prior Test baselevels are addressed. New features are not incorporated during this period. If major problems are observed in the final baselevel, it may be rebuilt. The duration is typically four to six weeks. After a final build is declared complete, there is a one-week window. This time is used to stage the ISOs and RPM packages for final distribution and begin the push to mirror sites.

After a Fedora Core distribution is released, new versions of packages are available from the yum repositories. In this context, the new packages are referred to as "updates."

Next issue: The final article in this series describes how Red Hat Enterprise Linux shares common goals with the Fedora Project. It will also tell how we tailor our Red Hat Enterprise Linux development model to meet the needs of enterprise customers.