Issue #5 March 2005

Red Hat Network in action

One site's journey

When Robert Spier began managing the servers that provide the perl.org website, just those two machines required more effort than he and his fellow system administrator Ask Bjoern Hansen were prepared to give. Anytime a vulnerability was identified, they worked manually to install the latest packages and resolve their inherent dependencies.

And then the infrastructure—housing mailing lists, bug trackers, and code repositories for the Perl community—grew.

"Once you get past two or three systems, replicating things becomes a real pain," lamented Spier, who now oversees the 17 Red Hat systems serving all of the specialized sites within the perl.org and pm.org (Perl Mongers) domains, as well as www.parrotcode.org. He began looking at tools to make this job easier. After some personal experimentation with Ximian Red Carpet, he settled on Red Hat Network (RHN) for the production environment more than two years ago.

"Having RHN tell us what's out-of-date is a big plus. We can update a machine in two or three clicks," Spier said, adding that RHN's simplicity and growing feature set won out over the Ximian solution. "After that point, Red Hat Network leapfrogged Red Carpet... We don't spend nearly as much time trying to figure out what package goes on what system. This lets us keep track easily. You don't have to worry about it. If I had to worry, it wouldn't be useful."

From there to here

This isn't too shabby for a management platform originally designed to help individual and small-business Red Hat Linux users update their systems. Although Red Hat had a full-scale management and monitoring service in mind, the Red Hat Network released in September 2000 (to coincide with Red Hat Linux 7) was limited in scope. When the RHN website was launched, for instance, it didn't support more than five systems per account.

Nevertheless, the response was overwhelming... literally. Roughly 50,000 systems were registered with RHN within its first month of operation.

"The initial infrastructure we rolled out with was brought to its knees by the popularity of the offering," remembers Cristian Gafton, one of the designers of Red Hat Network and now technical lead for the Fedora Project. "The feature requests poured in, from all sides, and all of the sudden the RHN engineering team was caught in a frenzy of reworking the infrastructure while at the same time addressing the critical feature requests that were coming from our more influential customers."

Two primary needs surfaced: One, the product had to be enhanced to suit larger deployments with added security and greatly increased scalability. And, two, all customers had to be assured quick and easy updates, including those for custom packages. The first set of requirements resulted in the Workgroup offering, while the second generated the RHN Proxy Server and RHN Satellite Server.

The Workgroup entitlement, now known as Management, arrived within a few months of RHN's launch. It enabled system grouping, captured additional details about systems, such as location, and supported actions performed across sets of systems simultaneously. Most important to large-scale customers, this service level also allowed multiple administrators to manage many systems within a single overarching account.

Then the team focused on its other priority: ensuring timely updates. To this end, the RHN Proxy Server and RHN Satellite Server were developed to bring software package repositories into customer environments. By caching the latest RPMs on separate RHN Servers behind corporate firewalls, Red Hat Network became unencumbered by Internet traffic and the requests of other users. In addition, the disconnected option of Satellite offered another layer of security.

Like the launch of RHN, these product releases garnered more eyes and resulted in more bugs and requests for enhancement. Subscriptions climbed to the half-million mark in no time. Within a year, the staff of six dedicated software engineers turned into 16 with support from virtually every department within the company.

In January 2003, RHN released added features aimed at multi-system management, as well as a more robust and intuitive website. Where once users were expected to change base channels for systems individually, for example, this association could now be made across the organization through the enhanced System Set Manager.

Later that year, RHN made its initial foray into Provisioning. Offering kickstarting, configuration management, snapshot-based rollbacks, custom system information, and an application programming interface (API), this was Red Hat Network's most ambitious release yet. (Refer to the Provisioning explored section for a description of these features.)

Finally, in late 2004, the RHN hosted service, Proxy Server, and Satellite Server provided: 1) a utility for bootstrapping client systems for Proxy and Satellite use; 2) the ability to immediately push actions from the Satellite (rather than wait for client systems to check in) and 3) an established RHN outage policy, as well as refinements to the kickstart and configuration management interfaces.

In that release, RHN Satellite Server customers were also given their first peak at Monitoring—running diagnostic probes on systems to track performance and prevent failures—in the form of a technology preview available freely to all Provisioning-entitled systems.

RHN in the enterprise

The advancements made since Red Hat Network's introduction have proven pivotal in readying it for large-scale deployments. Just ask Dirk Elmendorf, co-founder of and chief technology evangelist for Rackspace Managed Hosting, the fastest-growing managed hosting specialist in the world.

"It puts the management in crisis management," said Elmendorf, whose company oversees more than 6,000 servers with Red Hat Network.

"Before RHN, the patching process was cumbersome. It was very much a fire drill every time a vulnerability was discovered," he recounted, explaining the time to update the Rackspace infrastructure decreased from days to hours. "Before, you pushed something out and hoped nothing breaks. Now you know a patch is going out and it's going to work."

With customer concerns over security and Rackspace's own 100% uptime networking service level agreement (SLA), the company's army of system administrators previously spent a majority of its time merely keeping servers running and up-to-date. Then, when Red Hat Enterprise Linux 2.1 was released, Rackspace standardized on using Red Hat Network to manage those new systems in production, freeing up his administrators to handle individual customer requests and generally reducing the Linux systems' costs of ownership.

"Before RHN, every new machine played a role in staffing levels. The hiring process is no longer driven by patch management," Elmendorf explained. "It's made a dramatic difference in workflow. The Red Hat Network solution allows us to focus on our brand of service known as Fanatical Support®."

Getting updates

That's all well and good, but how do I receive updates?

At RHN's core is the Red Hat Update Agent, also commonly known by its command, up2date. Based upon the functionality of RPM Package Manager (RPM), the Red Hat Update Agent keeps track of a system's package list and can download and install new or updated packages whenever they arrive on the central RHN Servers.

All Update-entitled systems may use either the GUI or the command line version of up2date to retrieve the latest packages. Once the systems have been registered with RHN, simply issue the command up2date, or up2date --nox for systems not running the X Window System.

These systems may also have package updates scheduled through the RHN website. Once the system is registered and you're logged in, click Systems in the top navigation bar and then the name of the system. In the System Details page, click the Errata tab, select the updates to be made, and click Apply Errata.

For Management-entitled systems, administers may schedule package updates for multiple systems at once. In the RHN website, this is as simple as clicking the Errata tab in the top navigation bar, clicking the number of systems affected by the relevant Erratum, selecting the checkboxes of the affected systems you desire updated (or the top checkbox for all), and clicking Apply Errata.

"Another reason we use RHN (and Red Hat Enterprise Linux) is the trust factor," said Spier, of perl.org. "We know that the updates have been vetted by the Red Hat engineering staff."

And update is just the beginning of what you can do.

Smart management

Neal Mackanic is in the challenging position of enabling systems management in an intentionally distributed environment. As the computer scientist overseeing Lawrence Livermore National Laboratory's installation of RHN Satellite Server, he helps his fellow employees keep their systems secure without mandating specific update mechanisms.

"This is no cookie-cutter environment," said Mackanic, explaining the premier scientific center encourages diversity of system types to foster creativity. Scientists there run Windows, Macintosh and various flavors of UNIX and Linux on their desktops, while most servers tend to run UNIX and Linux.

About 800 Red Hat systems are now connected to the RHN Satellite Server at LLNL, with that number growing as more Linux users discover the management service and UNIX users migrate to Linux.

"I don't have to push it, just manage it," he said of his fellow scientists' reception to Red Hat Network.

Although the Satellite Server's enhanced security and rapid update times may have lured lab users, Mackanic took steps to ensure the integration went smoothly. He carefully crafted a process by which every Red Hat user in the laboratory might easily register their systems, have those systems assigned to the appropriate system groups, and obtain direct oversight of those systems, all while taking into consideration Lawrence Livermore's 12 departments broken down into dozens of organizational units.

Here's how it works:

  1. Create a system group for each of the lab's administrative domains, like Engineering using the organization administrator account.
  2. Figure 1. Creating the engineering system group
  3. Create an activation key of the same name.
  4. Figure 2. Creating the activation key
  5. Associate the key with the system group.
  6. Figure 3. Tying the new activation key to the engineering group
  7. Distribute the key character set to the relevant organizational unit, in this case the Engineering department.
  8. Have employees of that unit use the key to register new systems, using the command:
    rhnreg_ks --activationkey=<KEY_CHARACTER_SET>

    Note that Red Hat Enterprise Linux 2.1 systems use the --serialnumber option instead of --activationkey.

  9. Create accounts for the organizational units and grant them permissions to manage the appropriate system groups:
  10. Figure 4. Granting access to the engineering system group for the eng-admin user
  11. Step back and allow users to maintain their own systems.

Mackanic has found that creating a one-to-one correlation between system groups, activation keys, and administrative domains eases the introduction of new users while ensuring some consistency between departments. Keys and groups are shared across units, and user accounts are isolated to individuals.

He is streamlining this process further by initiating use of a bootstrap script now available to RHN Satellite Server and RHN Proxy Server customers. In addition to redirecting client systems to the Satellite or Proxy, this script can implement security measures, such as GPG keys and SSL certificates, and conduct post-registration configuration of the systems.

Provisioning explored

With Provisioning, users may develop kickstart profiles through the website, upload configuration files for safe keeping, revert systems to a previous instance, and provide any key-value pair about systems, which may then be queried.

This section walks you through some of these tasks.

Kickstart

Kickstarting is one of the most powerful functions of Linux, allowing the automated installation of an operating system on multiple systems using the input of a single text file. RHN seeks to alleviate the complexity of building the kickstart file by collecting its contents through a series of fields and menus on the RHN website.

To kickstart a machine in RHN:

  1. Create a distribution, or installation tree, for the kickstart. If you're on an RHN Satellite Server and desire a distribution matching a software channel already supplied by Red Hat, you may skip this step and select the distribution during creation of the kickstart profile.
  2. Upload any GPG and SSL keys to be included in the kickstart profile.
  3. Create a kickstart profile with an intuitive name and label and the initial configuration you desire, such as DHCP or static IP addressing.
  4. Once the initial profile exists, continue its configuration through the various tabs of the Kickstart Details page. Note the values included on the Options tab apply to the kickstart process itself, while the commands flagged on the Advanced Options subtab are included in the kickstart configuration file. On the subsequent tabs, you may include or exclude specific software packages, edit the %pre and %post scripts, add GPG and SSL keys, and identify IP addresses of systems to be presented this kickstart profile.
  5. After the profile is complete, you may use it to kickstart any Provisioning-entitled system in your account.

Configuration management

The configuration management interface allows you to store and share configuration files for your systems. Like software packages, configuration files may be placed in channels, either global configuration channels or system-specific configuration channels.

A global channel contains configuration files developed across your organization. These may well be applicable to multiple systems. A system-specific channel consists of local override configuration files tied to particular systems. These files take precedent over all other configurations. To manage a configuration file:

  1. In the Manage Config Channels interface, available to Configuration Administrators and Organization Administrators only, create a configuration channel with an appropriate name, label, and description.
  2. In the Configuration Channel Details page, either upload or create the files to be included in the channel.
  3. On Provisioning-entitled systems, install all rhncfg-* packages from the RHN Tools child software channel and create the /etc/sysconfig/rhn/allowed-actions/configfiles/ directory and relevant permissions file, if this hasn't already been done.
  4. Go to each system's System Details -> Channels -> Configuration -> Sandbox subtab to experiment with the files included in the new configuration channel.
  5. When satisfied with the files, use the parallel Config Channels subtab to subscribe the system to the configuration channel and assign it the rank corresponding to the order in which it should be applied.

Snapshot rollbacks

Every action that occurs on a Provisioning-entitled system—whether it's a package installation, an entitlement change, or anything in between generates a snapshot of that system as it existed prior to the change. You may then tag this snapshot and use it to revert the system's package profile, configuration files, and RHN settings to this state later. To do this:

  1. Conduct an action on a system, such as installing a package.
  2. Go to the System Details -> Snapshots tab and confirm the existence of the snapshot.
  3. Click the name of the snapshot reason and examine the various subtabs of the Snapshot Details page for associated changes.
  4. Go to the Snapshot Tags subtab and provide a meaningful phrase to describe the system change. This step is optional but recommended.
  5. Whenever you desire, return to the Snapshot Details page and click Rollback to Snapshot to completely revert the system to that state.

Custom system information

Although Red Hat Network has long collected software and hardware information about systems and allowed users to add notes to those system profiles, this hasn't met the needs of everyone. For instance, those notes are not clearly structured and cannot be searched. So Provisioning-entitled systems can now have any key-value pair registered against them that can then be queried, like so:

  1. In the Custom System Info page under Systems, create a new key, such as 'asset.'
  2. In the System Details -> Custom Info subtab, enter a value for the new key, such as 'ExampleCompany#456.'
  3. Go to the Advanced Search page and search for the value (in this case, '456') within the Custom Info field. The system will appear.

Tips & tricks

With Management and Provisioning entitlements, you have a plethora of poweful functions at your fingertips. And this sheer abundance can cause some confusion. This section seeks to offer hints gleaned from successful customers of Red Hat Network.

Creating and entitling the account

If you are a corporate customer, create a suitable account. Choose a login name that represents the company, rather than yourself. Similarly, the email address you provide should be a contact list that will enable RHN correspondence to make it to all relevant parties, regardless of an individual's time off or staffing change.

When obtaining RHN entitlements, be sure to purchase them using that corporate account. Entitlements bought with personal accounts are difficult to transfer.

Grouping systems

As the method of deployment Mackanic chose for LLNL shows, you want to create system groups that accurately reflect the means by which they'll be adminstered. In other words, you should break systems down into groups that make them easily manageable.

For some organizations, this means grouping systems by department. On the contrary, you may decide to categorize systems by function, the roles they serve in your infrastructure. Groups created with this in mind might be Web servers, application servers, and databases.

Another method is to establish groups based upon geography. These locales may be large or small, from cities and states all the way down to rack location. Still others manage their systems by platform or hardware type, such as Opteron or PowerEdge.

Finally, you may combine these methods to better isolate manageable groups within large infrastructures, such as databases—New York, and so on. Whatever method you choose, ensure it meshes with the duties of your system administrators and is used consistently.

Establishing administrators

Like system groups, RHN user accounts should represent your chosen method of administration. Although you may not create each account with a system group in mind, you should at least consider the roles the employee will serve.

Many companies choose to establish accounts based upon individual user information, such as email address. This greatly simplifies matching the actions of an account to a user. Others establish user accounts that correspond to system groups. This minimizes the burden of creating new accounts, deleting old ones, and reassigning permissions to address staffing changes.

For more details about these recommendations, consult the Corporate Account Practices Guide.

What the magic 8 says

Many longtime customers are now fully realizing the benefits of various Provisioning features. Mackanic, of Lawrence Livermore, is most impressed with the kickstart interface but also appreciates configuration management and the API.

Similarly, Spier at perl.org sees the advantages of snapshot replication, configuration management, and remote execution.

"If we have a catastrophic failure, we know we can get the box back up quickly because the configuration has been saved," said Spier, recalling a corrupt machine that led to a corrupt RPM database. "I love snapshots. They've saved our butts a few times."

Spier now plans to leverage an RHN Proxy Server to save bandwidth. Rackspace's Elmendorf, like Mackanic, still wants more granular permissions. But he says the API alone was worth the upgrade.

"We've got people that work with RHN systems," Elmendorf said. "When we've got our systems working with RHN systems, then it gets really exciting."

If history is any indicator, their wishes will be met. Red Hat Network has incorporated many features first floated by corporate users. For instance, the bootstrap script to speed client configuration that was included in the Provisioning release of RHN Satellite Server was suggested by Mackanic. And his request for RHN Notification Tool (or applet) support on Satellite was fulfilled in mid-2004.

"The engineering team has been interested in hearing about them," Mackanic said.

Red Hat Network development is now working to:

  • expand the RHN API to provide full access to every function of the website
  • support Red Hat Desktop and all other future Red Hat products
  • accommodate clustering, virtualization, and security-enhanced Linux

To find out more about Red Hat Network, refer to the online guides and release notes.

Entitlements

To accommodate its diverse set of customers—from small development shops to large, critical deployments—Red Hat Network offers various entitlements, or service levels:

Update entitlement

If you are a single-system user and not part of a larger organization, this entitlement is likely >ight for you. It:

  • provides simple, GUI-based updates, priority email notification, auditing and logging capabilities.
  • supports single-system management.
  • comes with all versions of Red Hat Enterprise Linux.

Management entitlement

If you are part of an organization that manages sets of systems, this may be the appropriate entitlement. It:

  • provides everything the Update entitlement does, as well as systems grouping, role-based administration, scheduled actions, an API access layer, and more.
  • facilitates comparisons of package sets across systems.
  • enables management of large-scale enterprises managing tens, hundreds, or even thousands of systems.

Provisioning entitlement

This entitlement is suited for organizations remotely overseeing vast infrastructures with a large number of administrators. It:

  • provides GUI-based kickstarts, snapshot rollbacks, and configuration file management.
  • allows remote command execution (with appropriate permissions set).
  • collects and enables searching upon completely customizable system information.

Environments

Similarly, these services can come to customers from three places: the central RHN Servers, an RHN Proxy Server, or an RHN Satellite Server.

Central RHN servers

This form, also known as RHN's hosted environment, is the most commonly used. Since the Red Hat Update Agent comes with Red Hat Enterprise Linux, and it's automatically configured to connect to rhn.redhat.com for updates, starting to use the central servers is as simple as activating your software and issuing the command up2date.

Figure 5. The standard Red Hat Network-hosted environment

All three entitlements are supported in the hosted environment. The hosted model is designed for companies with limited Red Hat Enterprise Linux deployments who would like to leverage the Red Hat Network infrastructure to store and manage their systems information and do not require a solution for custom content.

RHN Proxy Server

In this model, individual systems connect to a locally hosted (on the customer network) RHN Proxy Server. This machine aggregates all packages and performs updates locally. Although the Proxy communicates with the central RHN Servers via the Internet to synchronize and cache packages, the central RHN website is used to initiate actions.

Figure 6. Locally-hosted RHN Proxy server setup

The Proxy model is ideal for small companies that require a staging environment for custom or third-party content and who want to cache content locally to provide faster distribution to systems.

RHN Satellite Server

With an RHN Satellite Server, all RHN functionality is on the customer premises, allowing the customer greater functionality and customization. The RHN Satellite Server connects with RHN over the Internet only to download updates, and even this is optional since updates can be obtained in ISO form and then transferred to the Satellite.

Figure 7. RHN Satellite server setup

The Satellite model is the preferred architecture of enterprise customers with larger RHEL deployments who require enhanced Management and Provisioning features, and for those with stringent security policies requiring all system data be kept on the local network.

In addition, you may use some combination of these products to suit your infrastructure. Some larger customers have opted for RHN Proxy Servers to receive their updates from an RHN Satellite Server, completely closing the loop within the company and ensuring quick access to downloads for thousands of Linux systems.

Refer to redhat.com for a direct comparison of these entitlement levels and environments.

RHN Flash demo

Interactive Features Tour

Learn more about the benefits of Red Hat Network.
This interactive tour requires Macromedia Flash Player 6.

About the author

Clay Murphy has spent the past two years corralling changes in Red Hat Network into usable documentation. Before technical writing, he was foolish enough to practice newspaper journalism. His latest assignment brings him to the newly acquired Red Hat Directory Server and Certificate System products.