Live Kernel Patching Update

In the year since I first wrote about kpatch, Red Hat's live kernel patching project for Linux, we've been very busy.  Here are some of the highlights from the last year of live kernel patching development, and some clues about where we may be headed in the future.

Red Hat Enterprise Linux 7 Special Interest Group

In 2014, we kicked off a kpatch Special Interest Group (SIG) for users who are interested in trying out kpatch in a Red Hat Enterprise Linux 7 environment.  We've delivered kpatch fixes for several kernel CVEs, allowing users to easily apply fixes to their kernels immediately with no disruption or reboots necessary.

If you're a Red Hat Enterprise Linux customer and are interested in joining the kpatch SIG

, please contact your Red Hat Account Manager or Technical Account Manager (TAM) for more information on how to participate in this SIG.

Rapid Development

The kpatch development team worked at a feverish pace in 2014.  We fixed 95 issues and merged 376 pull requests.  We also grew into a strong community.  Some of my favorite highlights:

• The addition of support for safely patching data structures.
• A markedly improved percentage of patches that can be applied to a running kernel.  For example, in a recent test of 40 kernel CVE fixes, we were able to live patch all 40 of them, for a 100% patch compatibility rate!
• The porting of kpatch to many distributions, including Red Hat Enterprise Linux, Fedora, CentOS, Ubuntu, Debian, and Oracle Linux.
• The addition of support for patching kernel modules.
• The creation of a large integration test suite.
• The addition of support for the upstream livepatch project (more on this below).
• The incorporation of many stability, performance and usability improvements.
• And last but not least - the amazing contributions from 14 people, most of whom were not Red Hat employees!

To see some of the impressive things that kpatch can do, I encourage you to check out the following short demo from Seth Jennings...

http://www.youtube.com/watch?v=MLzM3lnx5Mc

LinuxCon North America Presentations

At LinuxCon North America In August 2014, I presented an introduction to kpatch entitled "kpatch: Have Your Security And Eat It Too!".

Also at LinuxCon, Hitachi's Masami Hiramatsu gave a presentation entitled "kpatch Without Stop Machine", where he presented a fantastic in-depth proposal of a performance improvement to kpatch.

Both talks resulted in a lot of thoughtful questions and excellent discussions.  The slides can be found here and here.

Upstream Collaboration

After we had matured kpatch to a point where we wanted to share it with a wider community, we learned that SUSE had also created a live kernel patching technology called kGraft.

At Red Hat, we have a deep understanding of the power of the open source development model.  We default to open.  So we began to have discussions with the kGraft team in order to try to figure out how to combine forces.

In April 2014, both teams met informally at the Collaboration Summit in Napa.  We mutually agreed that combining the projects somehow would be a good idea.  But we still had some technical hurdles to overcome before collaboration could be possible.

In October 2014, Red Hat's Steven Rostedt (maintainer of the Linux ftrace tracing facility) organized the Live Kernel Patching Microconference at the Linux Plumbers Conference in Düsseldorf. It was a great opportunity for all interested parties to come together face-to-face for some in-depth conversations about live kernel patching.

After much discussion, the kpatch and kGraft teams successfully worked out a plan for how to combine the two approaches into a single approach that would be suitable for merging into the upstream Linux kernel.

Live Kernel Patching Merged for Linux 4.0

As a result of the collaborative talks in Düsseldorf, Red Hat's Seth Jennings created a new kernel component called livepatch, a common base patching layer which is compatible with both kpatch and kGraft approaches. In November 2014, he submitted the first version of livepatch for review to the Linux kernel mailing list.

In February 2015, thanks to everyone's efforts over the last year and beyond, Linus Torvalds merged live kernel patching into upstream Linux!

This result is a tremendous success story which demonstrates the power of open source development. Two previously "competing" projects came together to build something greater than the sum of its parts. It's yet another example of how Red Hat is deeply committed to collaboration with the open source community.

The Future of kpatch and Live Kernel Patching

As you can see, there were some exciting developments related to kpatch and live kernel patching in the last year.  But there's still a lot of work left to do.

kpatch is a full software stack which includes both kernel and user application code. Our current focus is on continuing to work upstream to port the remaining kernel pieces (or some equivalent to them) into Linux.

We've already made significant progress on that front for Linux 4.0, which is due in April. It's slated to have all the functionality needed to be able to live patch the vast majority of kernel security fixes.

In future versions of Linux we hope to add support for changing function prototypes and data structures.

In parallel, the kpatch project will gradually move into maintenance mode as more of its functionality gets ported into upstream Linux.  Of course, that may be easier said than done.  Who knows what the next year will bring?  Stay tuned.