Linux patch management: How to back out a failed patch

Keeping servers up to date is one of a system administrator's primary responsibilities. However, updates do not always work the way you expect, so it's equally important that you know how to a) revert a patch to get the server back to the previous state and b) apply patches in subsets to get more flexibility.

This article focuses on operating system level patching done with the help of the package management tools YUM/DNF on Red Hat Enterprise Linux (RHEL). For this demonstration, a running RHEL 7.7 server is updated to RHEL 7.9 using yum, and then reverted back to the RHEL 7.7 version. It's always recommended to do periodic backups of your servers and the demonstrations provided in this article are for educational purposes only.

Check the server for pending patches/updates

Unless you're using a systems management tool like Red Hat Satellite, you need to test the servers for any updates manually.

For this article, consider a RHEL 7.7 server:

[root@rhel77 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)

Use the check-update command to non-interactively check for outstanding updates on your server:

[root@rhel77 ~]# yum check-update
Loaded plugins: product-id, search-disabled-repos, subscription-manager

NetworkManager.x86_64                                        1:1.18.4-3.el7                          rhel-7-server-rpms
NetworkManager-config-server.noarch                          1:1.18.4-3.el7                          rhel-7-server-rpms
NetworkManager-libnm.x86_64                                  1:1.18.4-3.el7                          rhel-7-server-rpms
NetworkManager-team.x86_64                                   1:1.18.4-3.el7                          rhel-7-server-rpms
kernel-tools-libs.x86_64                                     3.10.0-1127.19.1.el7                    rhel-7-server-rpms
kexec-tools.x86_64                                           2.0.15-43.el7                           rhel-7-server-rpms
kmod.x86_64                                                  20-28.el7                               rhel-7-server-rpms
kmod-libs.x86_64                                             20-28.el7                               rhel-7-server-rpms
kpartx.x86_64                                                0.4.9-131.el7                           rhel-7-server-rpms
krb5-libs.x86_64                                             1.15.1-46.el7                           rhel-7-server-rpms
….
….
….
systemd.x86_64                                               219-73.el7_8.9                          rhel-7-server-rpms
systemd-libs.x86_64                                          219-73.el7_8.9                          rhel-7-server-rpms
systemd-sysv.x86_64                                          219-73.el7_8.9                          rhel-7-server-rpms
tcpdump.x86_64                                               14:4.9.2-4.el7_7.1                      rhel-7-server-rpms
teamd.x86_64                                                 1.29-1.el7                              rhel-7-server-rpms
tuned.noarch                                                 2.11.0-8.el7                            rhel-7-server-rpms
tzdata.noarch                                                2020a-1.el7                             rhel-7-server-rpms
util-linux.x86_64                                            2.23.2-63.el7                           rhel-7-server-rpms
yum.noarch                                                   3.4.3-167.el7                           rhel-7-server-rpms
yum-utils.noarch                                             1.1.31-54.el7_8                         rhel-7-server-rpms
Obsoleting Packages
iwl7260-firmware.noarch                                      25.30.13.0-76.el7                       rhel-7-server-rpms
    iwl7265-firmware.noarch                                  22.0.7.0-72.el7                         installed     

[ You might also like: Red Hat Insights: Vulnerability management ]

When this article was written, RHEL 7.9 was the most recent version of RHEL 7 and doing a yum update would take this RHEL 7.7 server to RHEL 7.9:

[root@rhel77 ~]# yum -y update
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package NetworkManager.x86_64 1:1.18.0-5.el7 will be updated
---> Package NetworkManager.x86_64 1:1.18.8-1.el7 will be an update
---
Complete!

After a successful reboot, check the RHEL version to confirm that the server is updated to 7.9:

[root@rhel77 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)

Reverting the updates

First, check the history of the transaction with the yum command:

[root@rhel77 ~]# yum history
Loaded plugins: product-id, search-disabled-repos, subscription-manager
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
     8 | root <root>              | 2020-11-01 23:10 | I, O, U        |  157 EE

The Action(s) and Altered columns give information about what changes occurred with this transaction.

Action(s):

• I - New package Installed
• O - Package is Obsoleted
• U - Package is Updated

Altered:

• 157 packages were altered
• EE - There were some errors/warnings in the transaction

Using yum history packages-list shows the changes that happened from that package's point of view.

As systemd is the first process started in RHEL versions 7 and above, that package is protected with /etc/yum/protected.d/systemd.conf:

[root@rhel77 ~]# cat /etc/yum/protected.d/systemd.conf
systemd

Revert the last transaction with yum history undo:

[root@rhel77 ~]# yum history undo last
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Undoing transaction 8, from Sun Nov  1 23:10:56 2020
    Updated     NetworkManager-1:1.18.0-5.el7.x86_64                        @?rhel-7-server-rpms
    Update                     1:1.18.8-1.el7.x86_64                        @rhel-7-server-rpms
    Updated     NetworkManager-config-server-1:1.18.0-5.el7.noarch          @?rhel-7-server-rpms
    Update                                   1:1.18.8-1.el7.noarch          @rhel-7-server-rpms
    Updated     NetworkManager-libnm-1:1.18.0-5.el7.x86_64                  @?rhel-7-server-rpms
....
Resolving Dependencies
--> Running transaction check
…
--> Finished Dependency Resolution
Error: Trying to remove "systemd", which is protected

Caution: The following process is unsupported and untested. Resolve any issues that arise from the downgrade by recovering from a backup or reinstallation.

Move the yum configuration file, which protects systemd:

[root@rhel77 ~]# mv /etc/yum/protected.d/systemd.conf /etc/yum/protected.d/systemd.conf.bak

Re-run the yum history undo last command:

[root@rhel77 ~]# yum history undo last
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Undoing transaction 8, from Sun Nov  1 23:10:56 2020
....
Skipping the running kernel: kernel-3.10.0-1160.2.2.el7.x86_64
....

Dependencies Resolved

=======================================================================================================================
 Package                                    Arch       Version                           Repository               Size
=======================================================================================================================
Installing:
 iwl7265-firmware                           noarch     22.0.7.0-72.el7                   rhel-7-server-rpms      7.3 M
Removing:
 bc                                         x86_64     1.06.95-13.el7                    @rhel-7-server-rpms     215 k
 iwl7260-firmware                           noarch     25.30.13.0-79.el7                 @rhel-7-server-rpms      87 M
 linux-firmware                             noarch     20200421-79.git78c0348.el7        @rhel-7-server-rpms     394 M
Downgrading:
 NetworkManager                             x86_64     1:1.18.0-5.el7                    rhel-7-server-rpms      1.9 M
 NetworkManager-config-server               noarch     1:1.18.0-5.el7                    rhel-7-server-rpms      149 k
 NetworkManager-libnm    
…

Transaction Summary
=======================================================================================================================
Install      1 Package
Remove       3 Packages
Downgrade  152 Packages
…
Running transaction
  Installing : libgcc-4.8.5-39.el7.x86_64                                                                        1/308
  Installing : 1:grub2-common-2.02-0.80.el7.noarch                                                               2/308
  Installing : redhat-release-server-7.7-10.el7.x86_64                                                           3/308
warning: /etc/os-release saved as /etc/os-release.rpmsave
  Installing : setup-2.8.71-10.el7.noarch                                                                        4/308
warning: /etc/shadow created as /etc/shadow.rpmnew
  Installing : 1:grub2-pc-modules-2.02-0.80.el7.noarch   
....
yum-utils.noarch 0:1.1.31-52.el7                   

Complete!

Reboot the server

Check the version of RHEL and the kernel:

[root@rhel77 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)

[root@rhel77 ~]# rpm -q kernel
kernel-3.10.0-1062.el7.x86_64
kernel-3.10.0-1160.2.2.el7.x86_64

[root@rhel77 ~]# uname -r
3.10.0-1160.2.2.el7.x86_64

The server is using the latest kernel version from RHEL 7.9. You can change the default kernel using grub2-set-default command.

Apply the patch in subsets to get more flexibility of rollback

Pinning RHEL to a specific version is one way you can prevent the system from being updated to the latest minor version. Subscription Manager can be used to set the version of RHEL:

[root@rhel77 ~]# subscription-manager release --list
+-------------------------------------------+
          Available Releases
+-------------------------------------------+
7.0
7.1
7.2
7.3
7.4
7.5
7.6
7.7
7.8
7.9
7Server

Check the available updates and repo status before pinning the version:

[root@rhel77 ~]# yum update
...
Transaction Summary
=======================================================================================================================
Install    2 Packages (+2 Dependent packages)
Upgrade  152 Packages


[root@rhel77 ~]# subscription-manager repos --list-enabled
Repo ID:   rhel-7-server-rpms
Repo Name: Red Hat Enterprise Linux 7 Server (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/latest/$basearch/os
Enabled:   1

Pin the version to RHEL 7.8, clear the yum cache, and check the enabled repos:

[root@rhel77 ~]# subscription-manager release --set=7.8

[root@rhel77 ~]# yum clean all

[root@rhel77 ~]# subscription-manager repos --list-enabled
Repo ID:   rhel-7-server-rpms
Repo Name: Red Hat Enterprise Linux 7 Server (RPMs)
Repo URL:  https://cdn.redhat.com/content/dist/rhel/server/7/7.8/$basearch/os
Enabled:   1

It's evident from the Repo URL output above that it's now using 7.8 and not latest.

Run yum update and check whether there are changes to the updates available:

[root@rhel77 ~]# yum update
....
Transaction Summary
=======================================================================================================================
Install    2 Packages (+1 Dependent package)
Upgrade  136 Packages

Patching in subsets

The history option with yum uses the transaction to rollback/undo the operation. So, updating the packages as subsets gives the flexibility to roll them back. This is a better approach than updating the system as a whole.

Update the 'device-mapper' package with update device-mapper.

[root@rhel77 ~]# yum update device-mapper

Loaded plugins: product-id, search-disabled-repos, subscription-manager
Resolving Dependencies
--> Running transaction check
---> Package device-mapper.x86_64 7:1.02.158-2.el7 will be updated
--> Processing Dependency: device-mapper = 7:1.02.158-2.el7 for package: 7:device-mapper-libs-1.02.158-2.el7.x86_64
---> Package device-mapper.x86_64 7:1.02.170-6.el7 will be an update
--> Running transaction check
---> Package device-mapper-libs.x86_64 7:1.02.158-2.el7 will be updated
---> Package device-mapper-libs.x86_64 7:1.02.170-6.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================
 Package                        Arch               Version                        Repository                      Size
=======================================================================================================================
Updating:
 device-mapper                  x86_64             7:1.02.170-6.el7               rhel-7-server-rpms             297 k
Updating for dependencies:
 device-mapper-libs             x86_64             7:1.02.170-6.el7               rhel-7-server-rpms             325 k

Transaction Summary
=======================================================================================================================
Upgrade  1 Package (+1 Dependent package)

Total download size: 621 k
Is this ok [y/d/N]:

Check the history with yum history.

[root@rhel77 ~]# yum history
Loaded plugins: product-id, search-disabled-repos, subscription-manager
ID     | Login user               | Date and time    | Action(s)      | Altered
-------------------------------------------------------------------------------
     2 | root <root>              | 2020-11-02 15:25 | Update         |    2   
     1 | System <unset>           | 2020-09-14 04:25 | Erase          |    1 EE
history list

Get more specific details about transaction ID 2 using yum history info 2.

[root@rhel77 ~]# yum history info 2
Loaded plugins: product-id, search-disabled-repos, subscription-manager
Transaction ID : 2
Begin time     : Mon Nov  2 15:25:12 2020
Begin rpmdb    : 345:fc299a705f95203fda639ed1153803ad6771d7fb
End time       :            15:25:14 2020 (2 seconds)
End rpmdb      : 345:f8faef254573c0653a624c30ee393b08bb365ab2
User           : root <root>
Return-Code    : Success
Command Line   : update device-mapper
Transaction performed with:
    Installed     rpm-4.11.3-40.el7.x86_64                  installed
    Installed     subscription-manager-1.24.13-1.el7.x86_64 installed
    Installed     yum-3.4.3-163.el7.noarch                  installed
Packages Altered:
    Updated device-mapper-7:1.02.158-2.el7.x86_64      @?rhel-7-server-rpms
    Update                7:1.02.170-6.el7.x86_64      @rhel-7-server-rpms
    Updated device-mapper-libs-7:1.02.158-2.el7.x86_64 @?rhel-7-server-rpms
    Update                     7:1.02.170-6.el7.x86_64 @rhel-7-server-rpms
history info

Finally, undo the last transaction using yum history undo last.

[root@rhel77 ~]# yum history undo last

Loaded plugins: product-id, search-disabled-repos, subscription-manager
Undoing transaction 2, from Mon Nov  2 15:25:12 2020
    Updated device-mapper-7:1.02.158-2.el7.x86_64      @?rhel-7-server-rpms
    Update                7:1.02.170-6.el7.x86_64      @rhel-7-server-rpms
    Updated device-mapper-libs-7:1.02.158-2.el7.x86_64 @?rhel-7-server-rpms
    Update                     7:1.02.170-6.el7.x86_64 @rhel-7-server-rpms
Resolving Dependencies
--> Running transaction check
---> Package device-mapper.x86_64 7:1.02.158-2.el7 will be a downgrade
---> Package device-mapper.x86_64 7:1.02.170-6.el7 will be erased
---> Package device-mapper-libs.x86_64 7:1.02.158-2.el7 will be a downgrade
---> Package device-mapper-libs.x86_64 7:1.02.170-6.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================
 Package                        Arch               Version                        Repository                      Size
=======================================================================================================================
Downgrading:
 device-mapper                  x86_64             7:1.02.158-2.el7               rhel-7-server-rpms             294 k
 device-mapper-libs             x86_64             7:1.02.158-2.el7               rhel-7-server-rpms             322 k

Transaction Summary
=======================================================================================================================
Downgrade  2 Packages

Total download size: 616 k
Is this ok [y/d/N]:

[ A free guide from Red Hat: 5 steps to automate your business. ] 

It isn't the end

Being proactive is the best way to avoid rolling back a failed patch. Taking periodic operating system level and application-level backups is the recommended way to roll back the changes confidently. Read my Enable Sysadmin article on how to use ReaR for image-based backup and restore.

For more information on rolling back updates in Red Hat Enterprise Linux, be sure to check out the Red Hat Customer Portal.