Applies to SUSE Linux Enterprise Server 15 SP4

11 Live kernel patching with KLP #

This document describes the basic principles of the Kernel Live Patching (KLP) technology, and provides usage guidelines for the SLE Live Patching service.

KLP makes it possible to apply the latest security updates to Linux kernels without rebooting. This maximizes system uptime and availability, which is especially important for mission-critical systems.

The information provided in this document relates to the AMD64/Intel 64, POWER, and IBM Z architectures.

11.1 Advantages of Kernel Live Patching #

KLP offers several benefits.

Keeping a large number of servers automatically up to date is essential for organizations obtaining or maintaining certain compliance certifications. KLP can help achieve compliance, while reducing the need for costly maintenance windows.
Companies that work with service-level agreement contracts must guarantee a specific level of their system accessibility and uptime. Live patching makes it possible to patch systems without incurring downtime.
Since KLP is part of the standard system update mechanism, there is no need for specialized training or introduction of complicated maintenance routines.

11.2 Kernel Live Patching overview #

Kernel live patches are delivered as packages with modified code that are separate from the main kernel package. The live patches are cumulative, so the latest patch contains all fixes from the previous ones for the kernel package. Each kernel live package is tied to the exact kernel revision for which it is issued. The live patch package version number increases with every addition of fixes.

Note: Live patches and the running kernel

To determine the kernel patching status, use the klp -v patches command. The uname command's output does not change for patched kernels.

Important: Live patches compared to kernel updates

Live patches contain only critical fixes, and they do not replace regular kernel updates that require a reboot. Consider live patches as temporary measures that protect the kernel until a proper kernel update and a reboot are performed.

The diagram below illustrates the overall relationship between live patches and kernel updates. The list of CVEs and defect reports addressed by the currently active live patch can be viewed using the klp -v patches command.

It is possible to have multiple versions of the kernel package installed along with their live patches. These packages do not conflict. You can install updated kernel packages along with live patches for the running kernel. In this case, you may be prompted to reboot the system. Users with SLE Live Patching subscriptions are eligible for technical support as long as there are live patch updates for the running kernel (see Section 11.5.1, “Checking expiration date of the live patch”).

With KLP activated, every kernel update comes with a live patch package. This live patch does not contain any fixes and serves as a seed for future live patches for the corresponding kernel. These empty seed patches are called initial patches.

11.2.1 Kernel Live Patching scope #

The scope of SLE Live Patching includes fixes for SUSE Common Vulnerability Scoring System (CVSS; SUSE CVSS is based on the CVSS v3.0 system) level 7+ vulnerabilities and bug fixes related to system stability or data corruption. However, it may not be technically feasible to create live patches for all fixes that fall under the specified categories. SUSE therefore reserves the right to skip fixes in situations where creating a kernel live patch is not possible for technical reasons. Currently, over 95% of qualifying fixes are released as live patches. For more information on CVSS (the base for the SUSE CVSS rating), see Common Vulnerability Scoring System SIG.

11.2.2 Kernel Live Patching limitations #

KLP involves replacing functions and gracefully handling replacement of interdependent function sets. This is done by redirecting calls to old code to updated code in a different memory location. Changes in data structures make the situation more complicated, as the data remain in place and cannot be extended or reinterpreted. While there are techniques that allow indirect alteration of data structures, some fixes cannot be converted to live patches. In this situation, a system restart is the only way to apply the fixes.

11.3 Activating Kernel Live Patching using YaST #

To activate KLP on your system, you need to have active SLES and SLE Live Patching subscriptions. Visit SUSE Customer Center to check the status of your subscriptions and obtain a registration code for the SLE Live Patching subscription.

To activate Kernel Live Patching on your system, follow these steps:

Run the yast2 registration command and click Select Extensions.
Select SUSE Linux Enterprise Live Patching 15 in the list of available extensions and click Next.
Confirm the license terms and click Next.
Enter your SLE Live Patching registration code and click Next.
Check the Installation Summary and selected Patterns. The patterns Live Patching and SLE Live Patching Lifecycle Data should be automatically selected for installation along with additional packages to satisfy dependencies.
Click Accept to complete the installation. This will install the base Kernel Live Patching components on your system, the initial live patch, and the required dependencies.

11.4 Activating Kernel Live Patching from the command line #

To activate Kernel Live Patching, you need to have active SLES and SLES Live Patching subscriptions. Visit SUSE Customer Center to check the status of your subscriptions and obtain a registration code for the SLES Live Patching subscription.

Run sudo SUSEConnect --list-extensions. Note the exact activation command for SLES Live Patching. Example command output (abbreviated):

$ SUSEConnect --list-extensions
...
SUSE Linux Enterprise Live Patching 15 SP4 x86_64
Activate with: SUSEConnect -p sle-module-live-patching/15.4/x86_64 \
  -r ADDITIONAL REGCODE

Activate SLES Live Patching using the obtained command followed by -r LIVE_PATCHING_REGISTRATION_CODE, for example:
```
SUSEConnect -p sle-module-live-patching/15.4/x86_64 \
  -r LIVE_PATCHING_REGISTRATION_CODE
```
Install the required packages and dependencies using the command zypper install -t pattern lp_sles

At this point, the system has already been live-patched.

Here is how the process works behind the scenes: When the package installation system detects that there is an installed kernel that can be live-patched, and that there is a live patch for it in the software channel, the system selects the live patch for installation. The kernel then receives the live patch fixes as part of the package installation. The kernel gets live-patched even before the product installation is complete.

11.5 Performing Kernel Live Patching #

Kernel live patches are installed as part of regular system updates. However, there are several things you should be aware of.

The kernel is live-patched if a kernel-livepatch-* package has been installed for the running kernel. You can use the command zypper se --details kernel-livepatch-* to check what kernel live patch packages are installed on your system.
When the kernel-default package is installed, the update manager prompts you to reboot the system. To prevent this message from appearing, you can filter out kernel updates from the patching operation. This can be done by adding package locks with Zypper. SUSE Manager also makes it possible to filter channel contents (see Live Patching with SUSE Manager).
You can check patching status using the klp status command. To examine installed patches, run the klp -v patches command.
Keep in mind that while there may be multiple kernel packages installed on the system, only one of them is running at any given time. Similarly, there may be multiple live patch packages installed, but only one live patch is loaded into the kernel.
The active live patch is included in the initrd. This means that in case of an unexpected reboot, the system comes up with the live patch fixes applied, so there is no need to perform patching again.

11.5.1 Checking expiration date of the live patch #

Make sure that the lifecycle-data-sle-module-live-patching is installed, then run the zypper lifecycle command. You should see expiration dates for live patches in the Package end of support if different from product section of the output.

Every live patch receives updates for one year from the release of the underlying kernel package. The Maintained kernels, patch updates and lifecycle page allows you to check expiration dates based on the running kernel version without installing the product extension.

11.6 Troubleshooting Kernel Live Patching issues #

11.6.1 Manual patch downgrade #

If you find the latest live patch problematic, you can downgrade the currently installed live patch back to its previous version. We recommend performing patch downgrade before the system starts exhibiting issues. Keep in mind that a system with kernel warnings or kernel error traces in the system log may not be suitable for the patch downgrade procedure. If you are unsure whether the system meets the requirements for a patch downgrade, contact SUSE Technical Support for help.

Procedure 11.1: Manual patch downgrade #

Identify the running live patch using the klp -v patches command. You can see the currently running patch on the line starting with RPM:. For example:
```
RPM: kernel-livepatch-5_3_18-24_29-default-2-2.1.x86_64
```
The 5_3_18-24_29-default in the example above denotes the exact running kernel version.
Use the command zypper search -s kernel-livepatch-RUNNING_KERNEL_VERSION-default to search for previous versions of the patch. The command returns a list of available package versions. Keep in mind that for every new live patch package release, the version number increases by one. Make sure that you choose the version number one release lower than the current one.
Install the desired version with the command zypper in --oldpackage kernel-livepatch-RUNNING_KERNEL_VERSION-default=DESIRED_VERSION.