Debugging a kernel in QEMU/libvirt

This article was originally published on the Red Hat Customer Portal. The information may no longer be current.

A kernel bug announced on oss-security list claims to create a situation in which memory corruption can panic the system, by causing an integer used in determining the size of TCP send and receive buffers to be a negative value. Red Hat engineering sometimes backports security fixes and features from the current kernel, diverging the Red Hat Enterprise Linux kernel from upstream and causing some security issues to no longer apply. This blog post shows how to use live kernel debugging to determine if a system is at risk by this integer overflow flaw.

This walkthrough assumes that the reader has a Red Hat Enterprise Linux 7 guest system and basic knowledge of C programming.

Setting up the guest target to debug

The guest to be the target of the debugging session is a libvirt (or KVM/QEMU) style Virtual Machine. The guest virtual serial port should be mapped to the TCP port (TCP/1234) for use by GDB (the GNU Debugger).

Modifying the guest domain file

The virsh-edit command is intended to be a safe method of manipulating the raw XML that describes the guest, which is what we need to do in this circumstance. We need to configure the guest via the domain configuration file as there is no tickbox to enable what we need in virt-manager.

The first change is to set the XML namespace for QEMU, which sounds more complex than it is.

# virsh-edit your-virt-host-name

Find the domain directive and add the option xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'.

<domain type='kvm'
        xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0' >

Add a new qemu:commandline tag inside domain which will allow us to pass a parameter to QEMU for this guest when starting.

<domain type='kvm'
       xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0' >
     <qemu:commandline>
          <qemu:arg value='-s'/>
     </qemu:commandline>

Save the file and exit the editor. Some versions of libvirt may complain that the XML has invalid attributes; ignore this and save the file anyway. The libvirtd daemon does not need to be restarted. The guest will need to be destroyed and restarted if it is already running.

The -s parameter is an abbreviation of -gdb tcp::1234. If you have many guests needing debugging on different ports, or already have a service running on port 1234 on the host, you can set the port in the domain XML file as shown below:

  <qemu:commandline>
        <qemu:arg value='-gdb'/>
        <qemu:arg value='tcp::1235'/>
  </qemu:commandline>

If it is working, the QEMU process on the host will be listening on the port specified as shown below:

[root@target]# netstat -tapn | grep 1234
tcp        0      0 0.0.0.0:1234            0.0.0.0:*               LISTEN      11950/qemu-system-x

Change /etc/default/grub on the guest

The guest kernel will need to be booted with new parameters to enable KGDB debugging facilities. Add the values kgdboc=ttyS0,115200. In the system shown here, a serial console is also running on ttyS0 with no adverse effects.

Use the helpful grubby utility to apply these changes across all kernels.

# grubby --update-kernel=ALL --args="console=ttyS0,115200 kgdboc=ttyS0,115200"

Downloading debuginfo packages

The Red Hat Enterprise Linux kernel packages do not include debug symbols, as the symbols are stripped from binary files at build time. GDB needs those debug symbols to assist programmers when debugging. For more information on debuginfo see this segment of the Red Hat Enteprise Linux 7 developer guide.

RPM packages containing the name 'debuginfo' contain files with symbols. These packages can be downloaded from Red Hat using yum or up2date.

To download these packages on the guest:

# debuginfo-install --downloadonly kernel-3.10.0-327.el7

This should download two files in the current directory on the host for later extraction and use by GDB.

Copy these files from the guest to the host for the host GDB to use them. I choose ~/kernel-debug/ as a sane location for these files. Create the directory if it doesn't already exist.

# mkdir -p ~/kernel-debug
# scp yourlogin@guest:kernel*.rpm ~/kernel-debug/

The final step on the guest is to reboot the target. At this point the system should reboot with no change in behavior.

Preparing the host to debug

The system which runs the debugger doesn't need to be the host that contains the guest. The debugger system must be capable of making a connection to the guest running on the specified port (1234). In this example these commands will be run on the host which contains the virtual machine.

Installing GDB

Install GDB on the host using a package manager.

# sudo yum -y install gdb

Extracting files to be used from RPMs

When Red Hat builds the kernel it strips debugging symbols from the RPMs. This creates smaller downloads and uses less memory when running. The stripped packages are the well-known RPM packages named like kernel-3.10.0-327.el7.x86_64.rpm. The non-stripped debug information is stored in debuginfo rpms, like the ones downloaded earlier in this document by using debuginfo-install. They must match the exact kernel version and architecture being debugged on the guest to be of any use.

The target does not need to match the host system architecture or release version. The example below can extract files from RPMs on any system.

# cd ~/kernel-debug
# rpm2cpio kernel-debuginfo-3.10.0-327.el7.x86_64.rpm | cpio -idmv
# rpm2cpio kernel-debuginfo-common-3.10.0-327.el7.x86_64.rpm | cpio -idmv

This extracts the files within the packages into the current working directory as they would be on the intended file system. No scripts or commands within the RPMs are ran. These files are not installed and the system package management tools will not manage them. This allows them to be used on other architectures, releases or distributions.

• The unstripped kernel is the vmlinux file in ~/kernel-debug/usr/lib/debug/lib/modules/3.10.0-510.el7.x86_64/vmlinux
• The kernel source is in the directory ~/kernel-debug/usr/src/debug/kernel-3.10.0-327.el7/linux-3.10.0-327.el7.x86_64/

Connecting to the target system from the remote system

Start GDB with the text user interface, passing as a parameter the path to the unstripped kernel binary (vmlinux) running on the target system.

# gdb -tui ~/kernel-debug/var/lib/kernel-3.10.0-327.el7/boot/vmlinux

<gdb prelude shows here>

GDB must be told where to find the target system. Type the following into the GDB session:

set architecture i386:x86-64:intel
target remote localhost:1234
dir ~/kernel-debug/usr/src/debug/kernel-3.10.0-327.el7/linux-3.10.0-327.el7.x86_64/

Commands entered at the (gdb) prompt can be saved in ~/.gdbinit to reduce repetitive entry.

At this point, if all goes well, the system should be connected to the remote GDB session.

The story so far...

Congratulations, you've made it this far. If you've been following along you should have setup a GDB session to a system running in libvirt and be able to recreate and begin investigation into flaws.

Join the dark side as next time when we validate an integer promotion comparison flaw.