4 SystemTap—filtering and analyzing system data #

SystemTap usage is based on SystemTap scripts (*.stp). They tell SystemTap which type of information to collect, and what to do once that information is collected. The scripts are written in the SystemTap scripting language that is similar to AWK and C. For the language definition, see https://sourceware.org/systemtap/langref.pdf. A lot of useful example scripts are available from https://www.sourceware.org/systemtap/examples/.

The essential idea behind a SystemTap script is to name events, and to give them handlers. When SystemTap runs the script, it monitors for certain events. When an event occurs, the Linux kernel runs the handler as a sub-routine, then resumes. Thus, events serve as the triggers for handlers to run. Handlers can record specified data and print it in a certain manner.

The SystemTap language only uses a few data types (integers, strings, and associative arrays of these), and full control structures (blocks, conditionals, loops, functions). It has a lightweight punctuation (semicolons are optional) and does not need detailed declarations (types are inferred and checked automatically).

For more information about SystemTap scripts and their syntax, refer to Section 4.3, “Script syntax” and to the stapprobes and stapfuncs man pages, that are available with the systemtap-docs package.

Tapsets are a library of pre-written probes and functions that can be used in SystemTap scripts. When a user runs a SystemTap script, SystemTap checks the script's probe events and handlers against the tapset library. SystemTap then loads the corresponding probes and functions before translating the script to C. Like SystemTap scripts themselves, tapsets use the file name extension *.stp.

However, unlike SystemTap scripts, tapsets are not meant for direct execution. They constitute the library from which other scripts can pull definitions. Thus, the tapset library is an abstraction layer designed to make it easier for users to define events and functions. Tapsets provide aliases for functions that users could want to specify as an event. Knowing the proper alias is often easier than remembering specific kernel functions that may vary between kernel versions.

The main commands associated with SystemTap are stap and staprun. To execute them, you either need root privileges or must be a member of the stapdev or stapusr group.

For a list of options for each command, use --help. For details, refer to the stap and the staprun man pages.

To avoid giving root access to users solely to enable them to work with SystemTap, use one of the following SystemTap groups. They are not available by default on SUSE Linux Enterprise Server, but you can create the groups and modify the access rights accordingly. Also, adjust the permissions of the staprun command if the security implications are appropriate for your environment.

The following list gives an overview of the SystemTap main files and directories.

A SystemTap script can have multiple probes. They must be written in the following format:

probe EVENT {STATEMENTS}

Each probe has a corresponding statement block. This statement block must be enclosed in { } and contains the statements to be executed per event.

probe1 begin2
{3
   printf4 ("hello world\n")5
   exit ()6
}7

If your statement block holds several statements, SystemTap executes these statements in sequence—you do not need to insert special separators or terminators between multiple statements. A statement block can also be nested within another statement blocks. Generally, statement blocks in SystemTap scripts use the same syntax and semantics as in the C programming language.

SystemTap supports several built-in events.

The general event syntax is a dotted-symbol sequence. This allows a breakdown of the event namespace into parts. Each component identifier may be parameterized by a string or number literal, with a syntax like a function call. A component may include a * character, to expand to other matching probe points. A probe point may be followed by a ? character, to indicate that it is optional, and that no error should result if it fails to expand. Alternately, a probe point may be followed by a ! character to indicate that it is both optional and sufficient.

SystemTap supports multiple events per probe—they need to be separated by a comma (,). If multiple events are specified in a single probe, SystemTap will execute the handler when any of the specified events occur.

The events can be classified into the following categories:

probe timer.s(4)
{
   printf("hello world\n")
}

For detailed information about supported events, refer to the stapprobes man page. The See Also section of the man page also contains links to other man pages that discuss supported events for specific subsystems and components.

Each SystemTap event is accompanied by a corresponding handler defined for that event, consisting of a statement block.

function FUNCTION_NAME(ARGUMENTS) {STATEMENTS}
probe EVENT {FUNCTION_NAME(ARGUMENTS)}

printf ("1%s2(%d3) open\n4", execname(), pid())

[...]
vmware-guestd(2206) open
held(2360) open
[...]

foo = gettimeofday( )

global count_jiffies, count_ms
probe timer.jiffies(100) { count_jiffies ++ }
probe timer.ms(100) { count_ms ++ }
probe timer.ms(12345)
{
  hz=(1000*count_jiffies) / count_ms
  printf ("jiffies:ms ratio %d:%d => CONFIG_HZ=%d\n",
    count_jiffies, count_ms, hz)
  exit ()
  }