What's new in the Red Hat Satellite upgrade process

In this post we'll review a number of improvements that have been made to the Satellite upgrade process in the areas of technology, performance, and backend testing improvements and automation.

Over the last several releases the Satellite engineering and QE teams have been focused on making the Red Hat Satellite upgrade process much faster and more predictable.

The way the Satellite upgrades work has not changed—you will need to upgrade to each individual version of Satellite and you cannot skip versions. If you are running Satellite 6.4 and you want to go to Satellite 6.6, you will need to upgrade from Satellite 6.4 to Satellite 6.5, then to Satellite 6.6. These upgrades can be done back-to-back in the same outage window.

Technology improvements with Satellite upgrades

Starting with Satellite 6.3, the preferred path for upgrades is to use foreman-maintain. Foreman-maintain has been improved over the next several releases and changed to satellite-maintain as part of Satellite 6.5. Foreman-maintain / Satellite-maintain also includes a number of usability improvements such as backup and restore capability, support for external Satellite databases, check for installed hotfixes, and more.

We also added as part of Satellite 6.3 a number of pre- and post-upgrade checks which validate that the Satellite is in a known good state and is ready for the upgrade process to start prior to the upgrade. Then, after the upgrade, Satellite will be checked to be sure that it is in a fully operational status. These checks have been enhanced in following releases and have provided an excellent upgrade experience.

Locking was introduced in Satellite 6.6. This should prevent you from accidentally upgrading Satellite or related packages outside of satellite-maintain. For example, if you run “yum update -y” Satellite will not be updated to the latest release outside of the satellite-maintain process, avoiding the pre and post checks we put in place.

Performance improvements with Satellite upgrades

Each Satellite release includes a good number of performance improvements for various Satellite subsystems, and upgrades benefit quite a bit from several of these upgrades. While we don’t want this blog to turn into a large list of bugs, some of the big highlights are:

• Cleanup of old tasks in foreman-maintain (multiple BZs).

• Fixed sporadic ruby timeouts during upgrade in large Satellite environments (BZ1688317).

• Improved Import subscriptions task performance (BZ1683687).

• Easy upgrade of MongoDB storage engine to WiredTiger which improves MongoDB performance.

• Improved Candlepin API call performance (BZ1683708).

• Improved legacy mongo removal tasks (BZ1646741).

• Candlepin database consistency validated as part of pre-upgrade steps (BZ1520326).

• Improved satellite-tools-upgrade package for smooth upgrade in clients (BZ1541411).

Notable in the above is the MongoDB upgrade to WiredTiger. This is an optional upgrade that is available starting with Satellite 6.5. This improvement changes the on-disk format for how MongoDB stores its data from mmap to WiredTiger. 

There are some performance advantages of WiredTiger and we do recommend this change. The WiredTiger upgrade may take some time, so it isn't included as part of the installer and you do need to run it manually. The storage engine upgrade should be done just once, so if you happen to perform this upgrade in Satellite 6.5 you do not need to do it again in Satellite 6.6 or future releases.

Internal testing improvements with Satellite upgrades

Inside of Red Hat we have done a lot of changes that are generally not visible to you as an end customer. The Satellite quality engineering (QE) team added significant automation, enabling end to end upgrade testing from one version to the next, covering Satellite, Capsule and client upgrades. 

Automation was also added to test the post-upgrade experience to validate that common tasks are working as expected and that your data is valid after the upgrade completes. We also added a closed loop process that automated several bugzillas reported by customers, adding tests for these bugs that are now part of pre-upgrade / post-upgrade automated testing. 

We also use customer databases to test upgrades for every Satellite build. This gives us a very high expectation that Satellite upgrades will be successful in your environment.

Results

As a result of these changes in our upgrade process we have had a number of customers quickly adopting new releases as they become available. Some of our larger customers are seeing their upgrades completed in 90 minutes or less and they report that they're are finding the upgrades simple, easy, and reliable.

Next Steps

If you are running an older version of Satellite, we hope you will consider upgrading to the latest release very soon. As a reminder, Satellite 6.3 and earlier are end-of-life as of this writing. Red Hat recommends that you be on the latest version of Red Hat Satellite for the best customer experience.

If you are considering an upgrade but are concerned about having a poor upgrade experience, consider opening a proactive support case. If you have questions about the upgrade process, please open a support case or reach out to your account representative.