Upgrading from RHEL 7 to RHEL 8 with Leapp and BOOM

Red Hat Enterprise Linux was released a few weeks ago, and organizations may be considering upgrading from RHEL 7 to RHEL 8. In this post, I'll show how to carry out the upgrade process, minimize risks, and leverage the LVM snapshot feature to preserve the original root file system. Thanks to the snapshot, users should be able to go back to the RHEL 7 operating system in the event of any issues after the upgrade.

Note, while these steps have worked for us in a number of test scenarios, you should always try this with test systems similar to your own environment and have backups on hand before trying this in production.

Leapp & BOOM

Leapp is a CLI tool that helps users with the installation process and makes the in-place upgrade easier. 

Before moving to RHEL 8, you'll need to check the minimum version of RHEL 7 required to carry out the upgrade. Please check this knowledge base article ("Supported in-place paths for Red Hat Enterprise Linux" - login required) to learn more about supported versions. The final objective will be to migrate from RHEL 7.6 to the latest RHEL 8 minor version.

What Leapp does under the hood:

• Preparation of the upgrade process before the first reboot.

• Execution of the upgrade process in RAM disk.

• Execution of minor post-upgrade tasks after the upgrade from RAM disk.

BOOM is a utility available starting with RHEL 7.5, and we've covered this on the blog recently. This tool allows users to manage additional boot loader entries on the system. You can use it to create, delete, list, and modify auxiliary boot entries for system snapshots and images. The tool is provided by the lvm2-python-boom package.  

Getting ready for the upgrade

First of all, it is necessary to have your RHEL 7 system updated to the latest version (7.6), and check that it is both registered and subscribed. Once this requirement is accomplished, enable the optional and extra repositories:

[root@rhel7-test ~]# subscription-manager repos --enable rhel-7-server-optional-rpms --enable rhel-7-server-extras-rpms
Repository 'rhel-7-server-optional-rpms' is enabled for this system.
Repository 'rhel-7-server-extras-rpms' is enabled for this system.

Install the required packages to carry out the upgrade process:

[root@rhel7-test ~]# yum install -y leapp leapp-repository leapp-repository-deps lvm2-python-boom

Installing a sample httpd service

Since this is a test environment prepared to test the upgrade process, we're going to install an httpd service providing a sample HTTP page.

Install the required packages and do the necessary configurations.

[root@rhel7-test ~]# yum install httpd -y
[root@rhel7-test ~]# echo "Hello World" > /var/www/html/index.html
[root@rhel7-test ~]# systemctl enable --now httpd
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

Ensuring the service is up and running properly:

[root@rhel7-test ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:4f:2c:e4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.110/24 brd 192.168.122.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3f1f:81cf:9e0a:cf5a/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
[root@rhel7-test ~]# curl http://localhost
Hello World
[root@rhel7-test ~]# curl http://192.168.122.110
Hello World

Creating snapshots and BOOM entry

At this point, we're ready to create the snapshot that we'll use just in case it's necessary to roll back the whole upgrade process.

Listing the current Logical Volumes already present in this box we can observe that there are only two volumes. After creating the snapshot we'll see that there is a new volume present named "rollback".

The next step is to create the BOOM entry specifying the new snapshot rollback volume to this entry. As the process itself warns us, it's necessary to execute the "grub2-mkconfig > /boot/grub2/grub.cfg" command manually.

[root@rhel7-test ~]# lvs
 LV   VG Attr       LSize Pool Origin Data%  Meta% Move Log Cpy%Sync Convert
 root rhel -wi-ao---- 8.50g                                                    
 swap rhel -wi-ao---- 2.50g                                                    
[root@rhel7-test ~]# vgs
 VG   #PV #LV #SN Attr   VSize VFree
 rhel   2 2 0 wz--n- 23.99g 12.99g
                        

[root@rhel7-test ~]# lvcreate -s -L 9G -n rollback rhel/root
 Reducing COW size 9.00 GiB down to maximum usable size <8.54 GiB.
 Logical volume "rollback" created.

[root@rhel7-test ~]# lvs
 LV       VG Attr     LSize Pool Origin Data%  Meta% Move Log Cpy%Sync Convert
 rollback rhel swi-a-s--- <8.54g      root 0.01         
 root     rhel owi-aos---  8.50g                                        
 swap     rhel -wi-ao----  2.50g

[root@rhel7-test ~]# boom create --title "RHEL7.6 Snapshot - 2019 May 29th" --rootlv rhel/rollback
WARNING - Boom configuration not found in grub.cfg
WARNING - Run 'grub2-mkconfig > /boot/grub2/grub.cfg' to enable
Created entry with boot_id 94bb475:
 title RHEL7.6 Snapshot - 2019 May 29th
 machine-id aff60dca5fb14737858604118a273715
 version 3.10.0-957.el7.x86_64
 linux /vmlinuz-3.10.0-957.el7.x86_64
 initrd /initramfs-3.10.0-957.el7.x86_64.img
 options root=/dev/rhel/rollback ro rd.lvm.lv=rhel/rollback rhgb quiet

[root@rhel7-test ~]# grub2-mkconfig > /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-aff60dca5fb14737858604118a273715
Found initrd image: /boot/initramfs-0-rescue-aff60dca5fb14737858604118a273715.img
done

# boom list
BootID  Version                 Name RootDevice
cf9fd6d 3.10.0-957.el7.x86_64    Red Hat Enterprise Linux Server /dev/rhel/rollback

The upgrade process

Before starting with the upgrade, you'll need to manually add some data to the target system. You can download the files via access.redhat.com (a login is required). See the article for more information. The step is planned to be automated in the future. 

Copy the files as shown here:

[root@rhel7-test files]# pwd
/etc/leapp/files
[root@rhel7-test files]# mv /var/tmp/leapp-data3.tar.gz $PWD && tar -xf leapp-data3.tar.gz && rm -fR leapp-data3.tar.gz
rm: remove regular file 'leapp-data3.tar.gz'? y
[root@rhel7-test files]# ls -ltrh
total 1.7M
-rw-rw-r--. 1 1000 1000  339 Apr 12 11:40 repomap.csv
-rw-rw-r--. 1 1000 1000 1.6M Apr 26 00:09 pes-events.json

Finally, we can launch the upgrade using the Leapp tool.

[root@rhel7-test ~]# leapp upgrade --debug
2019-05-29 17:32:55.779 INFO     PID: 29577 leapp: Logging has been initialized
2019-05-29 17:32:55.800 INFO     PID: 29577 leapp.repository.common: A new repository 'common' is initialized at /etc/leapp/repos.d/common
2019-05-29 17:32:55.813 DEBUG    PID: 29577 leapp.repository.common: Scanning path /etc/leapp/repos.d/common
2019-05-29 17:32:55.823 DEBUG    PID: 29577 leapp.repository.common: Adding topic - topics/systeminfo.py
2019-05-29 17:32:55.835 INFO     PID: 29577 leapp.repository.system_upgrade_el7toel8: A new repository 'system_upgrade_el7toel8' is initialized at
…

You may find an error similar to the one shown here. As it explains in the report log, it's necessary to disable btrfs unloading the kernel module. The instructions on how to disable btrfs are listed in leapp-report.txt (sample below).

============================================================
                       ERRORS
============================================================
2019-05-30 12:48:18.266527 [ERROR] Actor: verify_check_results Message: Btrfs removed in the next major version
2019-05-30 12:48:18.288727 [ERROR] Actor: verify_check_results Message: Ending process due to errors found during checks, see /var/log/leapp/leapp-report.txt for detailed report.
============================================================
                    END OF ERRORS
============================================================

[root@rhel7-test ~]# cat /var/log/leapp/leapp-report.txt
Severity: high
Title: Btrfs removed in the next major version
Summary: The Btrfs file system was introduced as Technology Preview with the initial release of Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 7. As of versions 6.6 and 7.4 this technology has been deprecated and will be removed in next major version.
Remediation: Please consider migrating your Btrfs mount point(s) to a different filesystem before next upgrade attempt. If no Btrfs filesystem is in use, please unload btrfs kernel module running "# rmmod btrfs".
----------------------------------------
Severity: high
Title: Packages will not be installed
Summary: 2 packages will not be installed due to blacklisted repositories:
- rpcgen
- python3-pyxattr
----------------------------------------

Once the btrfs module is disabled, re-launch the upgrade process executing the same leapp command. When the process finishes reboot the system as prompted.

[root@rhel7-test ~]# leapp upgrade --debug
...
A reboot is required to continue. Please reboot your system.
[root@rhel7-test ~]# reboot

The upgrade process must be monitored from the console since the whole operation will be performed offline. Once the server is rebooted after the leapp process ends, a new option will be displayed and selected by default "RHEL Upgrade Initramfs". (This is the RAM disk from where the upgrade process will actually be performed.)

Once this initial phase within the upgrade process is completed, the system will reboot automatically. The new RHEL 8 GRUB entry is selected by default, so the system will boot from there. A few extra steps are required, like the SELinux relabeling. As stated, this will take some time depending on the system.

Checking the new upgraded system

As soon as the new system is available after the last reboot, we can check to see how the OS looks. What is the RHEL version? What is the kernel version? Finally, was the IP address preserved?

Let's do a quick check.

RHEL and kernel version, and IP address

​# ssh root@rhel7-test
root@rhel7-test's password:
Last login: Thu May 30 10:36:12 2019 from gateway
[root@rhel7-test ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.0 (Ootpa)
[root@rhel7-test ~]# uname -r
4.18.0-80.1.2.el8_0.x86_64

[root@rhel7-test ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
   link/ether 52:54:00:22:10:7a brd ff:ff:ff:ff:ff:ff
   inet 192.168.122.110/24 brd 192.168.122.255 scope global noprefixroute eth0
      valid_lft forever preferred_lft forever
   inet6 fe80::7bdb:98dd:8de1:8848/64 scope link noprefixroute
      valid_lft forever preferred_lft forever

HTTPD service

What is the status of the httpd service after upgrading to RHEL 8? Is everything working properly? Has the curl command retrieved the HTML example page we created during the first steps?

[root@rhel7-test ~]# systemctl status httpd
 httpd.service - The Apache HTTP Server
  Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Active: active (running) since Thu 2019-05-30 13:14:31 UTC; 18s ago
    Docs: man:httpd.service(8)
Main PID: 1285 (httpd)
  Status: "Running, listening on: port 80"
   Tasks: 213 (limit: 24017)
  Memory: 29.2M
  CGroup: /system.slice/httpd.service
          ├─1285 /usr/sbin/httpd -DFOREGROUND
          ├─1319 /usr/sbin/httpd -DFOREGROUND
          ├─1320 /usr/sbin/httpd -DFOREGROUND
          ├─1321 /usr/sbin/httpd -DFOREGROUND
          └─1322 /usr/sbin/httpd -DFOREGROUND

May 30 13:14:31 rhel7-test.example.com systemd[1]: Starting The Apache HTTP Server...
May 30 13:14:31 rhel7-test.example.com httpd[1285]: Server configured, listening on: port 80
May 30 13:14:31 rhel7-test.example.com systemd[1]: Started The Apache HTTP Server.
[root@rhel7-test ~]# curl http://localhost
Hello World
[root@rhel7-test ~]# curl http://192.168.122.110
Hello World

Yum and packages changes after the upgrade.

Let's check to see what changes have been made with the packages after upgrading the operating system. All the packages shown below have been upgraded to more recent versions--the ones that are available in RHEL 8 repositories.

# yum history
Updating Subscription Management repositories.
ID     | Command line             | Date and time | Action(s)      | Altered
-------------------------------------------------------------------------------
    1 | rhel-upgrade upgrade /tm | 2019-05-30 13:07 | D, E, I, O, U  | 559 EE

# rpm -qa|grep httpd
httpd-tools-2.4.37-11.module+el8.0.0+2969+90015743.x86_64
httpd-filesystem-2.4.37-11.module+el8.0.0+2969+90015743.noarch
redhat-logos-httpd-80.7-1.el8.noarch
httpd-2.4.37-11.module+el8.0.0+2969+90015743.x86_64

# rpm -qa|grep glibc
glibc-common-2.28-42.el8_0.1.x86_64
glibc-all-langpacks-2.28-42.el8_0.1.x86_64
glibc-2.28-42.el8_0.1.x86_64
glibc-locale-source-2.28-42.el8_0.1.x86_64
glibc-minimal-langpack-2.28-42.el8_0.1.x86_64

# rpm -qa|grep kernel
kernel-modules-4.18.0-80.1.2.el8_0.x86_64
kernel-4.18.0-80.1.2.el8_0.x86_64
kernel-tools-libs-4.18.0-80.1.2.el8_0.x86_64
kernel-3.10.0-957.el7.x86_64
kernel-workaround-0.1-1.el8.noarch
kernel-modules-extra-4.18.0-80.1.2.el8_0.x86_64
kernel-tools-4.18.0-80.1.2.el8_0.x86_64
kernel-core-4.18.0-80.1.2.el8_0.x86_64

After upgrading the operating system to RHEL 8, check to see if the following things are in place:

1. The RHEL version reported into the “/etc/redhat-release” is 8.0

2. The kernel version running is 4.18.0

3. The IP address is still configured and reachable from the network

4. The httpd server is running smoothly and the test HTML page is available

5. The versions of some packages have been updated to more recent versions

Rollback process

Hopefully, upgrading from the last minor version of RHEL 7 to the new version of RHEL 8 was a fairly easy process.

But, what happens if you detect an issue, or something unexpected comes up after upgrading to RHEL 8? If you recall, in the first steps a snapshot was created, just as a contingency method in the event of something going wrong.

To boot your recently upgraded RHEL 8 box from your previous RHEL 7 operating system, just restart the server and select the appropriate GRUB entry from the GRUB menu.

Checking the new upgraded system

As we checked before once the box has been upgraded to RHEL 8, check how the packages look. What are the versions that have been installed after booting from the snapshot? What is the version of RHEL already running? And what is the current IP address configured on the system?

RHEL and kernel version, and IP address.

​[root@rhel7-test ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)
[root@rhel7-test ~]# uname -r
3.10.0-957.el7.x86_64
[root@rhel7-test ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:4f:2c:e4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.110/24 brd 192.168.122.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3f1f:81cf:9e0a:cf5a/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

HTTPD service

[root@rhel7-test ~]# systemctl status httpd

● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-05-30 16:40:16 CEST; 1h 36min ago
     Docs: man:httpd(8)
           man:apachectl(8)
Main PID: 3524 (httpd)
   Status: "Total requests: 0; Current requests/sec: 0; Current traffic:   0 B/sec"
   CGroup: /system.slice/httpd.service
           ├─3524 /usr/sbin/httpd -DFOREGROUND
           ├─3570 /usr/sbin/httpd -DFOREGROUND
           ├─3571 /usr/sbin/httpd -DFOREGROUND
           ├─3572 /usr/sbin/httpd -DFOREGROUND
           ├─3574 /usr/sbin/httpd -DFOREGROUND
           └─3575 /usr/sbin/httpd -DFOREGROUND

May 30 16:40:16 rhel7-test.example.com systemd[1]: Starting The Apache HTTP Server...
May 30 16:40:16 rhel7-test.example.com systemd[1]: Started The Apache HTTP Server.
[root@rhel7-test ~]# curl http://localhost
Hello World
[root@rhel7-test ~]# curl http://192.168.122.110
Hello World

YUM and packages changes after the rollback

All the packages remain in their original versions, as there were before upgrading to RHEL 8. No track of the RHEL 8 upgrade in the yum history.

# rpm -qa|grep httpd
httpd-tools-2.4.6-89.el7_6.x86_64
httpd-2.4.6-89.el7_6.x86_64

# rpm -qa|grep glibc
glibc-common-2.17-260.el7.x86_64
glibc-2.17-260.el7.x86_64

#  rpm -qa|grep kernel
kernel-3.10.0-957.el7.x86_64
kernel-tools-3.10.0-957.el7.x86_64
kernel-tools-libs-3.10.0-957.el7.x86_64

# yum history
Loaded plugins: product-id, search-disabled-repos, subscription-manager
ID     | Login user               | Date and time | Action(s)      | Altered
-------------------------------------------------------------------------------
    3 | root <root>              | 2019-05-30 14:54 | Install       | 5
    2 | root <root>              | 2019-05-30 14:54 | Install       | 30
    1 | System <unset>           | 2019-05-30 14:11 | Install       | 340

As we checked before, when we were testing the OS recently upgraded to RHEL 8, all is working properly, this time after restarting and booting the OS from the snapshot that we made during the first steps.

RHEL and kernel versions are rolled back to the original versions, the same for all the other packages. The httpd service is running properly and the HTML test page is published and reachable via curl.

Final thoughts

Nowadays, thanks to these powerful tools, users can upgrade from RHEL 7 to RHEL 8. Leveraging the LVM snapshots feature makes the process safer, minimizes the risks of the upgrade process, and ensures you have available a valid and bootable copy of the root file system that the user can use in the event of any issues.

Finally, for a better understanding on the upgrade process using Leapp, you can find further information and limitations here.