Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
Applies to SUSE Linux Enterprise Server 15 SP1

8 Live Kernel Patching with KLP Edit source

This document describes the basic principles of the Kernel Live Patching (KLP) technology, and provides usage guidelines for the SLE Live Patching service.

KLP is a live patching technology for runtime patching of the Linux kernel, without stopping the kernel. This maximizes system uptime, and thus system availability, which is important for mission-critical systems. By enabling dynamic patching of the kernel, the technology also encourages users to install critical security updates without deferring them to a scheduled downtime.

Enabling KLP requires no special steps other than enabling the Live Patching service, and then applying the patches as they become available. The service is part of the normal software management system, and patches are installed (or removed) with the usual package management tools. There is no need to install or manually select special kernels.

A KLP patch is a kernel module, intended for replacing whole functions in the kernel. Kernel Live Patching primarily offers in-kernel infrastructure for integration of the patched code with the base kernel code at runtime.

The information provided in this document relates to the AMD64/Intel 64 and POWER architectures. KLP is supported on the Xen hypervisor.

8.1 Advantages of Kernel Live Patching Edit source

Kernel Live Patching using KLP is for quick emergency response, when serious vulnerabilities or system stability issues are known and should be fixed as quickly as possible. It is not used for scheduled updates where time is not critical.

Typical use cases for Kernel Live Patching include systems like memory databases with huge amounts of RAM, where boot-up times of 15 minutes or more are not uncommon, large simulations that need weeks or months without a restart, or infrastructure building blocks providing continuous service to many customers.

The main advantage of Kernel Live Patching is that it never requires stopping the kernel, not even for a short time period.

A KLP patch is a .ko kernel module in an RPM package. It is inserted into the kernel using the insmod command when the package is installed or updated. Kernel Live Patching replaces whole functions in the kernel, even if they are being executed. An updated KLP module can replace an existing patch if necessary.

Kernel Live Patching is also lean—it contains only a small amount of code, because it leverages other standard Linux technologies.

8.2 Low-level Function of Kernel Live Patching Edit source

Kernel Live Patching uses the ftrace infrastructure to perform patching. The following describes the implementation on the AMD64/Intel 64 architecture.

To patch a kernel function, Kernel Live Patching needs some space at the start of the function to insert a jump to a new function. This space is allocated during kernel compilation by GCC with function profiling turned on. In particular, a 5-byte call instruction is injected to the start of kernel functions. When such instrumented kernel is booting, profiling calls are replaced by 5-byte NOP (no operation) instructions.

After patching starts, the first byte is replaced by the INT3 (breakpoint) instruction. This ensures atomicity of the 5-byte instruction replacement. The other four bytes are replaced by the address to the new function. Finally, the first byte is replaced by the JMP (long jump) opcode.

Inter-processor non-maskable interrupts (IPI NMI) are used throughout the process to flush speculative decoding queues of other CPUs in the system. This allows switching to the new function without ever stopping the kernel, not even for a very short moment. The interruptions by IPI NMIs can be measured in microseconds and are not considered service interruptions as they happen while the kernel is running in any case.

Callers are never patched. Instead, the callee's NOPs are replaced by a JMP to the new function. JMP instructions remain forever. This takes care of function pointers, including in structures, and does not require saving any old data for the possibility of un-patching.

However, these steps alone would not be good enough: since the functions would be replaced non-atomically, a new fixed function in one part of the kernel could still be calling an old function elsewhere or vice versa. If the semantics of the function interfaces changed in the patch, chaos would ensue.

Thus, until all functions are replaced, Kernel Live Patching uses an approach based on trampolines and similar to RCU (read-copy-update), to ensure a consistent view of the world to each user space thread, kernel thread and kernel interrupt. A per-thread flag is set on each kernel entry and exit. This way, an old function would always call another old function and a new function always a new one. Once all processes have the "new universe" flag set, patching is complete, trampolines can be removed and the code can operate at full speed without performance impact other than an extra-long jump for each patched function.

8.3 Activation of SLE Live Patching Edit source

To activate SLE Live Patching on your system, follow these steps:

  1. Your SLES system must be registered. Register your system either during system installation, or after installation with the YaST Product Registration module (yast2 registration). If your SLES system is already registered, but SLE Live Patching is not yet activated, run the yast2 registration command and click Select Extensions.

  2. Select SUSE Linux Enterprise Live Patching 15 in the list of available extensions and click Next.

  3. Confirm the license terms and click Next.

  4. Enter your SLE Live Patching registration code and click Next.

  5. Check the Installation Summary and selected Patterns. The patterns Live Patching and SLE Live Patching Lifecycle Data should be automatically selected for installation, and there may be additional packages to satisfy dependencies.

  6. Click Accept to complete the installation. This will install the base Kernel Live Patching components on your system, the initial live patch, and any dependencies.

8.4 Installing and Removing Patches Edit source

This section describes how to find, install, and remove KLP patches.

8.4.1 Installing a KLP Patch Edit source

  1. Before installing new patches, run the klp status command to query current status, which must be ready and not in_progress. You cannot apply new patches until previous patch installations are completed. Invocations of the old kernel functions are not completely eliminated until all sleeping processes wake up and get out of the way. This can take a considerable amount of time. Sleeping processes that use the old kernel functions are not considered a security issue. (See Section 8.6, “Stuck Kernel Execution Threads” for information on managing prolonged in_progress states.)

  2. Use the normal package management system, zypper or YaST, to view and install available patches. The following example searches for available patches, and then installs the most recent patch. It is not necessary to install all patches in order; when there are multiple patches available install the most recent.

    tux > sudo zypper se kernel-livepatch 
    | kernel-livepatch-4_12_14-25_16-default  | Kernel live patch module  | package   
    | kernel-livepatch-4_12_14-25_19-default  | Kernel live patch module  | package   
    | kernel-livepatch-4_12_14-25_22-default  | Kernel live patch module  | package
    
    tux > sudo zypper in kernel-livepatch-4_12_14-25_22-default

8.4.2 Removing a KLP Patch Edit source

If you need to remove a KLP patch, use zypper just as you would for any other package. List your installed live patch packages by using zypper to search for kernel-livepatch:

tux > sudo zypper se kernel-livepatch 
  | kernel-livepatch-4_12_14-25_16-default  | Kernel live patch module  | package   
  | kernel-livepatch-4_12_14-25_19-default  | Kernel live patch module  | package   
i | kernel-livepatch-4_12_14-25_22-default  | Kernel live patch module  | package
  1. Remove the patch using zypper:

    tux > sudo zypper rm kernel-livepatch-4_12_14-25_22-default
  2. Wait for initrd to automatically rebuild, then reboot the machine.

8.5 The klp Tool Edit source

Several Kernel Live Patching management tasks can be simplified with the klp tool. The available commands are:

klp status

Displays the overall status of Kernel Live Patching (ready or in_progress).

klp patches

Displays the list of loaded KLP patches.

klp blocking

Lists processes that are preventing Kernel Live Patching from finishing. By default only the PIDs are listed. Specifying -v prints command lines if available. -vv displays stack traces.

For detailed information, see man klp.

8.6 Stuck Kernel Execution Threads Edit source

Kernel threads must be prepared to handle Kernel Live Patching. Third-party software may not support Kernel Live Patching, and may spawn kernel execution threads. These threads will block the patching process indefinitely. As an emergency measure, you may force the completion of the patching process without waiting for all execution threads to cross the safety checkpoint by writing 0 into /sys/kernel/livepatch/*/transition/ (replacing the asterisk wildcard with your file name). Consult SUSE support before performing this procedure.

8.7 Patch Lifecycle Edit source

Expiration dates of live patches can be accessed with zypper lifecycle. (Make sure that the package lifecycle-data-sle-module-live-patching is installed.)

When the expiration date of a patch is reached, no further live patches for this kernel version will be supplied. Plan an update of your kernel before the end of the live patch lifecycle period.

For details about zypper lifecycle, see the Showing Life Cycle Information in the Admin Guide.

8.8 Scope of Kernel Live Patching Technology Edit source

Kernel Live Patching is based on replacing functions. Data structure alteration can be accomplished only indirectly with Kernel Live Patching. As a result, changes to kernel data structure require special care and, if the change is too large, rebooting might be required. Kernel Live Patching also might not be able to handle situations where one compiler is used to compile the old kernel and another compiler is used for compiling the patch.

Because of the way Kernel Live Patching works, support for third-party modules that are spawning kernel threads is limited.

8.9 Scope of SLE Live Patching Edit source

Fixes for SUSE Common Vulnerability Scoring System (CVSS; SUSE CVSS is based on the CVSS v3.0 system) level 7+ vulnerabilities and bug fixes related to system stability or data corruption will be shipped in the scope of SLE Live Patching. It might not be possible to produce a live patch for all kinds of fixes fulfilling the above criteria. SUSE reserves the right to skip fixes where production of a kernel live patch is unviable because of technical reasons. For more information on CVSS, which is the base for the SUSE CVSS rating, see https://www.first.org/cvss/.

8.10 Interaction with the Support Processes Edit source

While resolving a technical difficulty with SUSE support, you may receive a Program Temporary Fix (PTF). PTFs may be issued for various packages including those forming the base of SLE Live Patching.

Kernel Live Patching PTFs complying with the conditions described in the previous section can be installed as usual and SUSE will ensure that the system in question does not need to be rebooted and that future live updates are applied cleanly.

PTFs issued for the base kernel disrupt the live patching process. First, installing the PTF kernel means a reboot as the kernel cannot be replaced as a whole at runtime. Second, another reboot is needed to replace the PTF with any regular maintenance updates for which the live patches are issued.

PTFs for other packages in SLE Live Patching can be treated like regular PTFs with the usual guarantees.

Print this page