8 Live kernel patching with KLP #
This document describes the basic principles of the Kernel Live Patching (KLP) technology, and provides usage guidelines for the SLE Live Patching service.
KLP makes it possible to apply the latest security updates to Linux kernels without rebooting. This maximizes system uptime and availability, which is especially important for mission-critical systems.
The information provided in this document relates to the AMD64/Intel 64, POWER, and IBM Z architectures.
8.1 Advantages of Kernel Live Patching #
KLP offers several benefits.
Keeping a large numbers of servers automatically up-to-date is essential for organizations obtaining or maintaining certain compliance certifications. KLP can help achieve compliance, while reducing the need for costly maintenance windows.
Companies that work with service-level agreement contracts must guarantee a specific level of their system accessibility and uptime. Live patching makes it possible to patch systems without incurring downtime.
Since KLP is part of the standard system update mechanism, there is no need for specialized training or introduction of complicated maintenance routines.
8.2 Kernel Live Patching overview #
Kernel live patches are delivered as packages with modified code that are separate from the main kernel package. The live patches are cumulative, so the latest patch contains all fixes from the previous ones for the kernel package. Each kernel live package is tied to the exact kernel revision for which it is issued. The live patch package version number increases with every addition of fixes.
After a live patch is applied, the
lp-HASH string is added to
the version of the running kernel as reported by the uname
-a command.
> uname -a
Linux sle15-sp3 5.3.18-150300.59.101-default #1 SMP \
Tue Nov 1 11:32:03 UTC 2022 (b2a976e/lp-cd28ef5) x86_64 x86_64 x86_64 GNU/Linux
To determine the kernel patching status, use the klp -v
patches command.
Live patches contain only critical fixes, and they do not replace regular kernel updates that require a reboot. Consider live patches as temporary measures that protect the kernel until a proper kernel update and a reboot is performed.
The diagram below illustrates the overall relationship between live patches
and kernel updates. The list of CVEs and defect reports addressed by the
currently-active live patch can be viewed using the klp -v
patches command.
It is possible to have multiple versions of the kernel package installed along with their live patches. These packages do not conflict. You can install updated kernel packages along with live patches for the running kernel. In this case, you may be prompted to reboot the system. Users with SLE Live Patching subscriptions are eligible for technical support as long as there are live patch updates for the running kernel (see Section 8.5.1, “Checking expiration date of the live patch”).
With KLP activated, every kernel update comes with a live patch package.
This live patch does not contain any fixes and serves as a seed for future
live patches for the corresponding kernel. These empty seed patches are
called initial patches.
8.2.1 Kernel Live Patching scope #
The scope of SLE Live Patching includes fixes for SUSE Common Vulnerability Scoring System (CVSS; SUSE CVSS is based on the CVSS v3.0 system) level 7+ vulnerabilities and bug fixes related to system stability or data corruption. However, it may not be technically feasible to create live patches for all fixes that fall under the specified categories. SUSE therefore reserves the right to skip fixes in situations where creating a kernel live patch is not possible for technical reasons. Currently, over 95% of qualifying fixes are released as live patches. For more information on CVSS (the base for the SUSE CVSS rating), see Common Vulnerability Scoring System SIG.
8.2.2 Kernel Live Patching limitations #
KLP involves replacing functions and gracefully handling replacement of interdependent function sets. This is done by redirecting calls to old code to updated code in a different memory location. Changes in data structures make the situation more complicated, as the data remain in place and cannot be extended or reinterpreted. While there are techniques that allow indirect alteration of data structures, some fixes cannot be converted to live patches. In this situation, a system restart is the only way to apply the fixes.
8.3 Activating Kernel Live Patching using YaST #
To activate KLP on your system, you need to have active SLES and SLE Live Patching subscriptions. Visit SUSE Customer Center to check the status of your subscriptions and obtain an registration code for the SLE Live Patching subscription.
To activate Kernel Live Patching on your system, follow these steps:
Run the
yast2 registrationcommand and click .Select in the list of available extensions and click .
Confirm the license terms and click .
Enter your SLE Live Patching registration code and click .
Check the and selected . The patterns
Live PatchingandSLE Live Patching Lifecycle Datashould be automatically selected for installation along with additional packages to satisfy dependencies.Click to complete the installation. This will install the base Kernel Live Patching components on your system, the initial live patch, and the required dependencies.
8.4 Activating Kernel Live Patching from the command line #
To activate Kernel Live Patching, you need to have an active SLES and SLES Live Patching subscriptions. Visit SUSE Customer Center to check the status of your subscriptions and obtain an registration code for the SLES Live Patching subscription.
Run
sudo SUSEConnect --list-extensions. Note the exact activation command for SLES Live Patching. Example command output (abbreviated):$ SUSEConnect --list-extensions ... SUSE Linux Enterprise Live Patching 15 SP2 x86_64 Activate with: SUSEConnect -p sle-module-live-patching/15.2/x86_64 \ -r ADDITIONAL REGCODEActivate SLES Live Patching using the obtained command followed by
-r LIVE_PATCHING_REGISTRATION_CODE, for example:SUSEConnect -p sle-module-live-patching/15.2/x86_64 \ -r LIVE_PATCHING_REGISTRATION_CODE
Install the required packages and dependencies using the command
zypper install -t pattern lp_sles
At this point, the system has already been live-patched.
Here is how the process works behind the scenes: When the package-installation system detects that there is an installed kernel that can be live-patched, and that there is a live patch for it in the software channel, the system selects the live patch for installation. The kernel then receives the live patch fixes as part of the package installation. The kernel gets live-patched even before the product installation is complete.
8.5 Performing Kernel Live Patching #
Kernel live-patches are installed as part of regular system updates. However, there are several things you should be aware of.
The kernel is live-patched if a kernel-livepatch-* package has been installed for the running kernel. You can use the command
zypper se --details kernel-livepatch-*to check what kernel live-patch packages are installed on your system.When the kernel-default package is installed, the update manager prompts you to reboot the system. To prevent this message from apparing, you can filter out kernel updates from the patching operation. This can be done by adding package locks with Zypper. SUSE Manager also makes it possible to filter channel contents (see Live Patching with SUSE Manager).
You can check patching status using the
klp statuscommand. To examine installed patches, run theklp -v patchescommand.Keep in mind that while there may be multiple kernel packages installed on the system, only one of them is running at any given time. Similarly, there may be multiple live patch packages installed, but only one live patch is loaded into the kernel.
The active live patch is included in the
initrd. This means that in case of an unexpected reboot, the system comes up with the live patch fixes applied, so there is no need to perform patching again.
8.5.1 Checking expiration date of the live patch #
Make sure that the
lifecycle-data-sle-module-live-patching is installed,
then run the zypper lifecycle command. You should see
expiration dates for live patches in the Package end of support if
different from product section of the output.
Every live patch receives updates for one year from the release of the underlying kernel package. The Maintained kernels, patch updates and lifecycle page allows you to check expiration dates based on the running kernel version without installing the product extension.
8.6 Troubleshooting Kernel Live Patching issues #
8.6.1 Manual patch downgrade #
If you find the latest live patch problematic, you can downgrade the currently installed live patch back to its previous version. We recommend performing patch downgrade before the system starts exhibiting issues. Keep in mind that a system with kernel warnings or kernel error traces in the system log may not be suitable for the patch downgrade procedure. If you are unsure whether the system meets the requirements for a patch downgrade, contact SUSE Technical Support for help.
Identify the running live patch using the
klp -v patchescommand. You can see the currently running patch on the line starting withRPM:. For example:RPM: kernel-livepatch-5_3_18-24_29-default-2-2.1.x86_64
The
5_3_18-24_29-defaultin the example above denotes the exact running kernel version.Use the command
zypper search -s kernel-livepatch-RUNNING_KERNEL_VERSION-defaultto search for previous versions of the patch. The command returns a list of available package versions. Keep in mind that for every new live patch package release the version number increases by 1. Make sure that you choose the version number one release lower than the current one.Install the desired version with the command
zypper in --oldpackage kernel-livepatch-RUNNING_KERNEL_VERSION-default=DESIRED_VERSION.