8 Live Kernel Patching with KLP #

Kernel Live Patching using KLP is for quick emergency response, when serious vulnerabilities or system stability issues are known and should be fixed as quickly as possible. It is not used for scheduled updates where time is not critical.

Typical use cases for Kernel Live Patching include systems like memory databases with huge amounts of RAM, where boot-up times of 15 minutes or more are not uncommon, large simulations that need weeks or months without a restart, or infrastructure building blocks providing continuous service to many customers.

The main advantage of Kernel Live Patching is that it never requires stopping the kernel, not even for a short time period.

A KLP patch is a .ko kernel module in an RPM package. It is inserted into the kernel using the insmod command when the package is installed or updated. Kernel Live Patching replaces whole functions in the kernel, even if they are being executed. An updated KLP module can replace an existing patch if necessary.

Kernel Live Patching is also lean—it contains only a small amount of code, because it leverages other standard Linux technologies.

Kernel Live Patching uses the ftrace infrastructure to perform patching. The following describes the implementation on the AMD64/Intel 64 architecture.

To patch a kernel function, Kernel Live Patching needs some space at the start of the function to insert a jump to a new function. This space is allocated during kernel compilation by GCC with function profiling turned on. In particular, a 5-byte call instruction is injected to the start of kernel functions. When such instrumented kernel is booting, profiling calls are replaced by 5-byte NOP (no operation) instructions.

After patching starts, the first byte is replaced by the INT3 (breakpoint) instruction. This ensures atomicity of the 5-byte instruction replacement. The other four bytes are replaced by the address to the new function. Finally, the first byte is replaced by the JMP (long jump) opcode.

Inter-processor non-maskable interrupts (IPI NMI) are used throughout the process to flush speculative decoding queues of other CPUs in the system. This allows switching to the new function without ever stopping the kernel, not even for a very short moment. The interruptions by IPI NMIs can be measured in microseconds and are not considered service interruptions as they happen while the kernel is running in any case.

Callers are never patched. Instead, the callee's NOPs are replaced by a JMP to the new function. JMP instructions remain forever. This takes care of function pointers, including in structures, and does not require saving any old data for the possibility of un-patching.

However, these steps alone would not be good enough: since the functions would be replaced non-atomically, a new fixed function in one part of the kernel could still be calling an old function elsewhere or vice versa. If the semantics of the function interfaces changed in the patch, chaos would ensue.

Thus, until all functions are replaced, Kernel Live Patching uses an approach based on trampolines and similar to RCU (read-copy-update), to ensure a consistent view of the world to each user space thread, kernel thread and kernel interrupt. A per-thread flag is set on each kernel entry and exit. This way, an old function would always call another old function and a new function always a new one. Once all processes have the "new universe" flag set, patching is complete, trampolines can be removed and the code can operate at full speed without performance impact other than an extra-long jump for each patched function.

To activate SLE Live Patching on your system, follow these steps:

This section describes how to find, install, and remove KLP patches.

> sudo zypper se kernel-livepatch
  | kernel-livepatch-4_12_14-25_16-default  | Kernel live patch module  | package   
  | kernel-livepatch-4_12_14-25_19-default  | Kernel live patch module  | package   
i | kernel-livepatch-4_12_14-25_22-default  | Kernel live patch module  | package

Several Kernel Live Patching management tasks can be simplified with the klp tool. The available commands are:

For detailed information, see man klp.

Kernel threads must be prepared to handle Kernel Live Patching. Third-party software may not support Kernel Live Patching, and may spawn kernel execution threads. These threads will block the patching process indefinitely. As an emergency measure, you may force the completion of the patching process without waiting for all execution threads to cross the safety checkpoint by writing 0 into /sys/kernel/livepatch/*/transition/ (replacing the asterisk wildcard with your file name). Consult SUSE support before performing this procedure.

Expiration dates of live patches can be accessed with zypper lifecycle. (Make sure that the package lifecycle-data-sle-module-live-patching is installed.)

When the expiration date of a patch is reached, no further live patches for this kernel version will be supplied. Plan an update of your kernel before the end of the live patch lifecycle period.

For details about zypper lifecycle, see the Showing Life Cycle Information in the Admin Guide.

Kernel Live Patching is based on replacing functions. Data structure alteration can be accomplished only indirectly with Kernel Live Patching. As a result, changes to kernel data structure require special care and, if the change is too large, rebooting might be required. Kernel Live Patching also might not be able to handle situations where one compiler is used to compile the old kernel and another compiler is used for compiling the patch.

Because of the way Kernel Live Patching works, support for third-party modules that are spawning kernel threads is limited.

Fixes for SUSE Common Vulnerability Scoring System (CVSS; SUSE CVSS is based on the CVSS v3.0 system) level 7+ vulnerabilities and bug fixes related to system stability or data corruption will be shipped in the scope of SLE Live Patching. It might not be possible to produce a live patch for all kinds of fixes fulfilling the above criteria. SUSE reserves the right to skip fixes where production of a kernel live patch is unviable because of technical reasons. For more information on CVSS, which is the base for the SUSE CVSS rating, see https://www.first.org/cvss/.

While resolving a technical difficulty with SUSE support, you may receive a Program Temporary Fix (PTF). PTFs may be issued for various packages including those forming the base of SLE Live Patching.

Kernel Live Patching PTFs complying with the conditions described in the previous section can be installed as usual and SUSE will ensure that the system in question does not need to be rebooted and that future live updates are applied cleanly.

PTFs issued for the base kernel disrupt the live patching process. First, installing the PTF kernel means a reboot as the kernel cannot be replaced as a whole at runtime. Second, another reboot is needed to replace the PTF with any regular maintenance updates for which the live patches are issued.

PTFs for other packages in SLE Live Patching can be treated like regular PTFs with the usual guarantees.