Red Hat Enterprise Linux was released a few weeks ago, and organizations may be considering upgrading from RHEL 7 to RHEL 8. In this post, I'll show how to carry out the upgrade process, minimize risks, and leverage the LVM snapshot feature to preserve the original root file system. Thanks to the snapshot, users should be able to go back to the RHEL 7 operating system in the event of any issues after the upgrade.
Note, while these steps have worked for us in a number of test scenarios, you should always try this with test systems similar to your own environment and have backups on hand before trying this in production.
Leapp & BOOM
Leapp is a CLI tool that helps users with the installation process and makes the in-place upgrade easier.
Before moving to RHEL 8, you'll need to update to the latest RHEL 7 minor release (which at the time of writing this is 7.6). The final objective will be to migrate from RHEL 7.6 to the latest RHEL 8 minor version.
What Leapp does under the hood:
Preparation of the upgrade process before the first reboot.
Execution of the upgrade process in RAM disk.
Execution of minor post-upgrade tasks after the upgrade from RAM disk.
BOOM is a utility available starting with RHEL 7.5, and we've covered this on the blog recently. This tool allows users to manage additional boot loader entries on the system. You can use it to create, delete, list, and modify auxiliary boot entries for system snapshots and images. The tool is provided by the lvm2-python-boom package.
Getting ready for the upgrade
First of all, it is necessary to have your RHEL 7 system updated to the latest version (7.6), and check that it is both registered and subscribed. Once this requirement is accomplished, enable the optional and extra repositories:
[root@rhel7-test ~]# subscription-manager repos --enable rhel-7-server-optional-rpms --enable rhel-7-server-extras-rpms Repository 'rhel-7-server-optional-rpms' is enabled for this system. Repository 'rhel-7-server-extras-rpms' is enabled for this system.
Install the required packages to carry out the upgrade process:
[root@rhel7-test ~]# yum install -y leapp leapp-repository leapp-repository-deps lvm2-python-boom
Installing a sample httpd service
Since this is a test environment prepared to test the upgrade process, we're going to install an httpd service providing a sample HTTP page.
Install the required packages and do the necessary configurations.
[root@rhel7-test ~]# yum install httpd -y [root@rhel7-test ~]# echo "Hello World" > /var/www/html/index.html [root@rhel7-test ~]# systemctl enable --now httpd Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.
Ensuring the service is up and running properly:
[root@rhel7-test ~]# ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:4f:2c:e4 brd ff:ff:ff:ff:ff:ff inet 192.168.122.110/24 brd 192.168.122.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever inet6 fe80::3f1f:81cf:9e0a:cf5a/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@rhel7-test ~]# curl http://localhost Hello World [root@rhel7-test ~]# curl http://192.168.122.110 Hello World
Creating snapshots and BOOM entry
At this point, we're ready to create the snapshot that we'll use just in case it's necessary to roll back the whole upgrade process.
Listing the current Logical Volumes already present in this box we can observe that there are only two volumes. After creating the snapshot we'll see that there is a new volume present named "rollback".
The next step is to create the BOOM entry specifying the new snapshot rollback volume to this entry. As the process itself warns us, it's necessary to execute the "grub2-mkconfig > /boot/grub2/grub.cfg" command manually.
[root@rhel7-test ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root rhel -wi-ao---- 8.50g swap rhel -wi-ao---- 2.50g [root@rhel7-test ~]# vgs VG #PV #LV #SN Attr VSize VFree rhel 2 2 0 wz--n- 23.99g 12.99g [root@rhel7-test ~]# lvcreate -s -L 9G -n rollback rhel/root Reducing COW size 9.00 GiB down to maximum usable size <8.54 GiB. Logical volume "rollback" created. [root@rhel7-test ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert rollback rhel swi-a-s--- <8.54g root 0.01 root rhel owi-aos--- 8.50g swap rhel -wi-ao---- 2.50g [root@rhel7-test ~]# boom create --title "RHEL7.6 Snapshot - 2019 May 29th" --rootlv rhel/rollback WARNING - Boom configuration not found in grub.cfg WARNING - Run 'grub2-mkconfig > /boot/grub2/grub.cfg' to enable Created entry with boot_id 94bb475: title RHEL7.6 Snapshot - 2019 May 29th machine-id aff60dca5fb14737858604118a273715 version 3.10.0-957.el7.x86_64 linux /vmlinuz-3.10.0-957.el7.x86_64 initrd /initramfs-3.10.0-957.el7.x86_64.img options root=/dev/rhel/rollback ro rd.lvm.lv=rhel/rollback rhgb quiet [root@rhel7-test ~]# grub2-mkconfig > /boot/grub2/grub.cfg Generating grub configuration file ... Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img Found linux image: /boot/vmlinuz-0-rescue-aff60dca5fb14737858604118a273715 Found initrd image: /boot/initramfs-0-rescue-aff60dca5fb14737858604118a273715.img done # boom list BootID Version Name RootDevice cf9fd6d 3.10.0-957.el7.x86_64 Red Hat Enterprise Linux Server /dev/rhel/rollback
The upgrade process
Before starting with the upgrade, you'll need to manually add some data to the target system. You can download the files via access.redhat.com (a login is required). See the article for more information. The step is planned to be automated in the future.
Copy the files as shown here:
[root@rhel7-test files]# pwd /etc/leapp/files [root@rhel7-test files]# mv /var/tmp/leapp-data3.tar.gz $PWD && tar -xf leapp-data3.tar.gz && rm -fR leapp-data3.tar.gz rm: remove regular file 'leapp-data3.tar.gz'? y [root@rhel7-test files]# ls -ltrh total 1.7M -rw-rw-r--. 1 1000 1000 339 Apr 12 11:40 repomap.csv -rw-rw-r--. 1 1000 1000 1.6M Apr 26 00:09 pes-events.json
Finally, we can launch the upgrade using the Leapp tool.
[root@rhel7-test ~]# leapp upgrade --debug 2019-05-29 17:32:55.779 INFO PID: 29577 leapp: Logging has been initialized 2019-05-29 17:32:55.800 INFO PID: 29577 leapp.repository.common: A new repository 'common' is initialized at /etc/leapp/repos.d/common 2019-05-29 17:32:55.813 DEBUG PID: 29577 leapp.repository.common: Scanning path /etc/leapp/repos.d/common 2019-05-29 17:32:55.823 DEBUG PID: 29577 leapp.repository.common: Adding topic - topics/systeminfo.py 2019-05-29 17:32:55.835 INFO PID: 29577 leapp.repository.system_upgrade_el7toel8: A new repository 'system_upgrade_el7toel8' is initialized at …
You may find an error similar to the one shown here. As it explains in the report log, it's necessary to disable btrfs unloading the kernel module. The instructions on how to disable btrfs are listed in
leapp-report.txt (sample below).
============================================================ ERRORS ============================================================ 2019-05-30 12:48:18.266527 [ERROR] Actor: verify_check_results Message: Btrfs removed in the next major version 2019-05-30 12:48:18.288727 [ERROR] Actor: verify_check_results Message: Ending process due to errors found during checks, see /var/log/leapp/leapp-report.txt for detailed report. ============================================================ END OF ERRORS ============================================================ [root@rhel7-test ~]# cat /var/log/leapp/leapp-report.txt Severity: high Title: Btrfs removed in the next major version Summary: The Btrfs file system was introduced as Technology Preview with the initial release of Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 7. As of versions 6.6 and 7.4 this technology has been deprecated and will be removed in next major version. Remediation: Please consider migrating your Btrfs mount point(s) to a different filesystem before next upgrade attempt. If no Btrfs filesystem is in use, please unload btrfs kernel module running "# rmmod btrfs". ---------------------------------------- Severity: high Title: Packages will not be installed Summary: 2 packages will not be installed due to blacklisted repositories: - rpcgen - python3-pyxattr ----------------------------------------
Once the btrfs module is disabled, re-launch the upgrade process executing the same leapp command. When the process finishes reboot the system as prompted.
[root@rhel7-test ~]# leapp upgrade --debug ... A reboot is required to continue. Please reboot your system. [root@rhel7-test ~]# reboot
The upgrade process must be monitored from the console since the whole operation will be performed offline. Once the server is rebooted after the leapp process ends, a new option will be displayed and selected by default "RHEL Upgrade Initramfs". (This is the RAM disk from where the upgrade process will actually be performed.)
Once this initial phase within the upgrade process is completed, the system will reboot automatically. The new RHEL 8 GRUB entry is selected by default, so the system will boot from there. A few extra steps are required, like the SELinux relabeling. As stated, this will take some time depending on the system.
Checking the new upgraded system
As soon as the new system is available after the last reboot, we can check to see how the OS looks. What is the RHEL version? What is the kernel version? Finally, was the IP address preserved?
Let's do a quick check.
RHEL and kernel version, and IP address
# ssh root@rhel7-test root@rhel7-test's password: Last login: Thu May 30 10:36:12 2019 from gateway [root@rhel7-test ~]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.0 (Ootpa) [root@rhel7-test ~]# uname -r 4.18.0-80.1.2.el8_0.x86_64 [root@rhel7-test ~]# ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 52:54:00:22:10:7a brd ff:ff:ff:ff:ff:ff inet 192.168.122.110/24 brd 192.168.122.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever inet6 fe80::7bdb:98dd:8de1:8848/64 scope link noprefixroute valid_lft forever preferred_lft forever
What is the status of the httpd service after upgrading to RHEL 8? Is everything working properly? Has the curl command retrieved the HTML example page we created during the first steps?
[root@rhel7-test ~]# systemctl status httpd httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2019-05-30 13:14:31 UTC; 18s ago Docs: man:httpd.service(8) Main PID: 1285 (httpd) Status: "Running, listening on: port 80" Tasks: 213 (limit: 24017) Memory: 29.2M CGroup: /system.slice/httpd.service ├─1285 /usr/sbin/httpd -DFOREGROUND ├─1319 /usr/sbin/httpd -DFOREGROUND ├─1320 /usr/sbin/httpd -DFOREGROUND ├─1321 /usr/sbin/httpd -DFOREGROUND └─1322 /usr/sbin/httpd -DFOREGROUND May 30 13:14:31 rhel7-test.example.com systemd: Starting The Apache HTTP Server... May 30 13:14:31 rhel7-test.example.com httpd: Server configured, listening on: port 80 May 30 13:14:31 rhel7-test.example.com systemd: Started The Apache HTTP Server. [root@rhel7-test ~]# curl http://localhost Hello World [root@rhel7-test ~]# curl http://192.168.122.110 Hello World
Yum and packages changes after the upgrade.
Let's check to see what changes have been made with the packages after upgrading the operating system. All the packages shown below have been upgraded to more recent versions--the ones that are available in RHEL 8 repositories.
# yum history Updating Subscription Management repositories. ID | Command line | Date and time | Action(s) | Altered ------------------------------------------------------------------------------- 1 | rhel-upgrade upgrade /tm | 2019-05-30 13:07 | D, E, I, O, U | 559 EE # rpm -qa|grep httpd httpd-tools-2.4.37-11.module+el8.0.0+2969+90015743.x86_64 httpd-filesystem-2.4.37-11.module+el8.0.0+2969+90015743.noarch redhat-logos-httpd-80.7-1.el8.noarch httpd-2.4.37-11.module+el8.0.0+2969+90015743.x86_64 # rpm -qa|grep glibc glibc-common-2.28-42.el8_0.1.x86_64 glibc-all-langpacks-2.28-42.el8_0.1.x86_64 glibc-2.28-42.el8_0.1.x86_64 glibc-locale-source-2.28-42.el8_0.1.x86_64 glibc-minimal-langpack-2.28-42.el8_0.1.x86_64 # rpm -qa|grep kernel kernel-modules-4.18.0-80.1.2.el8_0.x86_64 kernel-4.18.0-80.1.2.el8_0.x86_64 kernel-tools-libs-4.18.0-80.1.2.el8_0.x86_64 kernel-3.10.0-957.el7.x86_64 kernel-workaround-0.1-1.el8.noarch kernel-modules-extra-4.18.0-80.1.2.el8_0.x86_64 kernel-tools-4.18.0-80.1.2.el8_0.x86_64 kernel-core-4.18.0-80.1.2.el8_0.x86_64
After upgrading the operating system to RHEL 8, check to see if the following things are in place:
The RHEL version reported into the "/etc/redhat-release" is 8.0
The kernel version running is 4.18.0
The IP address is still configured and reachable from the network
The httpd server is running smoothly and the test HTML page is available
The versions of some packages have been updated to more recent versions
Hopefully, upgrading from the last minor version of RHEL 7 to the new version of RHEL 8 was a fairly easy process.
But, what happens if you detect an issue, or something unexpected comes up after upgrading to RHEL 8? If you recall, in the first steps a snapshot was created, just as a contingency method in the event of something going wrong.
To boot your recently upgraded RHEL 8 box from your previous RHEL 7 operating system, just restart the server and select the appropriate GRUB entry from the GRUB menu.
Checking the new upgraded system
As we checked before once the box has been upgraded to RHEL 8, check how the packages look. What are the versions that have been installed after booting from the snapshot? What is the version of RHEL already running? And what is the current IP address configured on the system?
RHEL and kernel version, and IP address.
[root@rhel7-test ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.6 (Maipo) [root@rhel7-test ~]# uname -r 3.10.0-957.el7.x86_64 [root@rhel7-test ~]# ip addr show eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:4f:2c:e4 brd ff:ff:ff:ff:ff:ff inet 192.168.122.110/24 brd 192.168.122.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever inet6 fe80::3f1f:81cf:9e0a:cf5a/64 scope link noprefixroute valid_lft forever preferred_lft forever
[root@rhel7-test ~]# systemctl status httpd ● httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2019-05-30 16:40:16 CEST; 1h 36min ago Docs: man:httpd(8) man:apachectl(8) Main PID: 3524 (httpd) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec" CGroup: /system.slice/httpd.service ├─3524 /usr/sbin/httpd -DFOREGROUND ├─3570 /usr/sbin/httpd -DFOREGROUND ├─3571 /usr/sbin/httpd -DFOREGROUND ├─3572 /usr/sbin/httpd -DFOREGROUND ├─3574 /usr/sbin/httpd -DFOREGROUND └─3575 /usr/sbin/httpd -DFOREGROUND May 30 16:40:16 rhel7-test.example.com systemd: Starting The Apache HTTP Server... May 30 16:40:16 rhel7-test.example.com systemd: Started The Apache HTTP Server. [root@rhel7-test ~]# curl http://localhost Hello World [root@rhel7-test ~]# curl http://192.168.122.110 Hello World
YUM and packages changes after the rollback
All the packages remain in their original versions, as there were before upgrading to RHEL 8. No track of the RHEL 8 upgrade in the
# rpm -qa|grep httpd httpd-tools-2.4.6-89.el7_6.x86_64 httpd-2.4.6-89.el7_6.x86_64 # rpm -qa|grep glibc glibc-common-2.17-260.el7.x86_64 glibc-2.17-260.el7.x86_64 # rpm -qa|grep kernel kernel-3.10.0-957.el7.x86_64 kernel-tools-3.10.0-957.el7.x86_64 kernel-tools-libs-3.10.0-957.el7.x86_64 # yum history Loaded plugins: product-id, search-disabled-repos, subscription-manager ID | Login user | Date and time | Action(s) | Altered ------------------------------------------------------------------------------- 3 | root <root> | 2019-05-30 14:54 | Install | 5 2 | root <root> | 2019-05-30 14:54 | Install | 30 1 | System <unset> | 2019-05-30 14:11 | Install | 340
As we checked before, when we were testing the OS recently upgraded to RHEL 8, all is working properly, this time after restarting and booting the OS from the snapshot that we made during the first steps.
RHEL and kernel versions are rolled back to the original versions, the same for all the other packages. The httpd service is running properly and the HTML test page is published and reachable via
Nowadays, thanks to these powerful tools, users can upgrade from RHEL 7 to RHEL 8. Leveraging the LVM snapshots feature makes the process safer, minimizes the risks of the upgrade process, and ensures you have available a valid and bootable copy of the root file system that the user can use in the event of any issues.