5 Kernel probes #
Kernel probes are a set of tools to collect Linux kernel debugging and performance information. Developers and system administrators usually use them either to debug the kernel, or to find system performance bottlenecks. The reported data can then be used to tune the system for better performance.
You can insert these probes into any kernel routine, and specify a handler to be invoked after a particular break-point is hit. The main advantage of kernel probes is that you no longer need to rebuild the kernel and reboot the system after you make changes in a probe.
To use kernel probes, you typically need to write or obtain a specific
kernel module. Such modules include both the init and
the exit function. The init function (such as
register_kprobe()
) registers one or more probes,
while the exit function unregisters them. The registration function
defines where the probe will be inserted and
which handler will be called after the probe is hit.
To register or unregister a group of probes at one time, you can use
relevant
register_<PROBE_TYPE>probes()
or
unregister_<PROBE_TYPE>probes()
functions.
Debugging and status messages are typically reported with the
printk
kernel routine.
printk
is a kernel space equivalent of a
user space printf
routine. For more information
on printk
, see
Logging
kernel messages. Normally, you can view these messages by
inspecting the output of the systemd
journal (see
Kapitel 21, journalctl
: Abfragen des systemd
-Journals). For more information on log files, see
Chapter 3, System log files.
5.1 Supported architectures #
Kernel probes are fully implemented on the following architectures:
x86
AMD64/Intel 64
Arm
POWER
Kernel probes are partially implemented on the following architectures:
IA64 (does not support probes on instruction
slot1
)sparc64 (return probes not yet implemented)
5.2 Types of kernel probes #
There are three types of kernel probes: Kprobes,
Jprobes, and Kretprobes.
Kretprobes are sometimes called return
probes. You can find source code examples of all three type of
probes in the Linux kernel. See the directory
/usr/src/linux/samples/kprobes/
(package
kernel-source
).
5.2.1 Kprobes #
Kprobes can be attached to any instruction in the Linux kernel. When Kprobes is registered, it inserts a break-point at the first byte of the probed instruction. When the processor hits this break-point, the processor registers are saved, and the processing passes to Kprobes. First, a pre-handler is executed, then the probed instruction is stepped, and, finally a post-handler is executed. The control is then passed to the instruction following the probe point.
5.2.2 Jprobes #
Jprobes is implemented through the Kprobes mechanism. It is
inserted on a function's entry point and allows direct access to the
arguments of the function which is being probed. Its handler routine
must have the same argument list and return value as the probed
function. To end it, call the jprobe_return()
function.
When a jprobe is hit, the processor registers are saved, and the
instruction pointer is directed to the jprobe handler routine. The
control then passes to the handler with the same register contents as the
function being probed. Finally, the handler calls the
jprobe_return()
function, and switches the
control back to the control function.
In general, you can insert multiple probes on one function. Jprobe is, however, limited to only one instance per function.
5.2.3 Return probe #
Return probes are also implemented through Kprobes. When the
register_kretprobe()
function is called, a
kprobe is attached to the entry of the probed function.
After hitting the probe, the kernel probes mechanism saves the probed
function return address and calls a user-defined return handler. The
control is then passed back to the probed function.
Before you call register_kretprobe()
, you need
to set a maxactive
argument, which specifies
how many instances of the function can be probed at the same time. If
set too low, you will miss a certain number of probes.
5.3 Kprobes API #
The programming interface of Kprobes consists of functions which are used to register and unregister all used kernel probes, and associated probe handlers. For a more detailed description of these functions and their arguments, see the information sources in Section 5.5, “More information”.
register_kprobe()
Inserts a break-point on a specified address. When the break-point is hit, the
pre_handler
andpost_handler
are called.register_jprobe()
Inserts a break-point in the specified address. The address needs to be the address of the first instruction of the probed function. When the break-point is hit, the specified handler is run. The handler should have the same argument list and return type as the probed.
register_kretprobe()
Inserts a return probe for the specified function. When the probed function returns, a specified handler is run. This function returns 0 on success, or a negative error number on failure.
unregister_kprobe()
,unregister_jprobe()
,unregister_kretprobe()
Removes the specified probe. You can use it any time after the probe has been registered.
register_kprobes()
,register_jprobes()
,register_kretprobes()
Inserts each of the probes in the specified array.
unregister_kprobes()
,unregister_jprobes()
,unregister_kretprobes()
Removes each of the probes in the specified array.
disable_kprobe()
,disable_jprobe()
,disable_kretprobe()
Disables the specified probe temporarily.
enable_kprobe()
,enable_jprobe()
,enable_kretprobe()
Temporarily enables disabled probes.
5.4 debugfs
Interface #
In recent Linux kernels, the Kprobes instrumentation uses the
kernel's debugfs
interface. It can list all
registered probes and globally switch all probes on or off.
5.4.1 Listing registered kernel probes #
The list of all currently registered probes is in the
/sys/kernel/debug/kprobes/list
file.
saturn.example.com:~ # cat /sys/kernel/debug/kprobes/list c015d71a k vfs_read+0x0 [DISABLED] c011a316 j do_fork+0x0 c03dedc5 r tcp_v4_rcv+0x0
The first column lists the address in the kernel where the probe is
inserted. The second column prints the type of the probe:
k
for kprobe, j
for jprobe, and
r
for return probe. The third column specifies the
symbol, offset and optional module name of the probe. The following
optional columns include the status information of the probe. If the
probe is inserted on a virtual address which is not valid anymore, it is
marked with [GONE]
. If the probe is temporarily
disabled, it is marked with [DISABLED]
.
5.4.2 Globally enabling/disabling kernel probes #
The /sys/kernel/debug/kprobes/enabled
file
represents a switch with which you can globally and forcibly turn on or
off all the registered kernel probes. To turn them off, simply enter
#
echo "0" > /sys/kernel/debug/kprobes/enabled
on the command line as root
. To turn them on again, enter
#
echo "1" > /sys/kernel/debug/kprobes/enabled
Note that this way you do not change the status of the probes. If a
probe is temporarily disabled, it will not be enabled automatically but
will remain in the [DISABLED]
state after entering
the latter command.
5.5 More information #
To learn more about kernel probes, look at the following sources of information:
Thorough but more technically oriented information about kernel probes is in
/usr/src/linux/Documentation/trace/kprobes.txt
(packagekernel-source
).Examples of all three types of probes (together with related
Makefile
) are in the/usr/src/linux/samples/kprobes/
directory (packagekernel-source
).In-depth information about Linux kernel modules and
printk
kernel routine can be found at The Linux Kernel Module Programming Guide