18 Using systemd-coredump
to Debug Application Crashes #
systemd-coredump
collects and displays kernel core dumps, for analyzing
application crashes. When a process crashes (or all processes belonging to an
application), its default is to log the core dump to the systemd
journal,
including a backtrace if possible, and to store the core dump in a file in
/var/lib/systemd/coredump
. You also have the option to
examine the dump file with other tools such as gdb
or
crash
(see Section 17.8, “Analyzing the Crash Dump”).
There is an option to not store core dumps, but only log them in the journal,
which may be useful to minimize the collection and storage of sensitive
information.
18.1 Use and Configuration #
systemd-coredump
is enabled and ready to run by default. The default
configuration is in /etc/systemd/coredump.conf
:
[Coredump] #Storage=external #Compress=yes #ProcessSizeMax=2G #ExternalSizeMax=2G #JournalSizeMax=767M #MaxUse= #KeepFree=
Size units are B, K, M, G, T, P, and E. ExternalSizeMax
also supports a value of infinity
.
The following example shows how to use Vim for simple testing, by creating a segfault to generate journal entries and a core dump.
Enable the
debuginfo-pool
anddebuginfo-update
repositoriesInstall vim-debuginfo and libebl1
Launch
vim testfile
and type a few charactersGet the PID and generate a segfault:
tux >
ps ax | grep vim 2345 pts/3 S+ 0:00 vim testfileroot #
kill -s SIGSEGV 2345List your core dumps, then examine them:
root #
coredumpctl TIME PID UID GID SIG PRESENT EXE Wed 2019-11-12 11:56:47 PST 2345 1000 100 11 * /bin/vimroot #
coredumpctl info PID: 2345 (vim) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Wed 2019-11-12 11:58:05 PST Command Line: vim testfile Executable: /bin/vim Control Group: /user.slice/user-1000.slice/session-1.scope Unit: session-1.scope Slice: user-1000.slice Session: 1 Owner UID: 1000 (tux) Boot ID: b5c251b86ab34674a2222cef102c0c88 Machine ID: b43c44a64696799b985cafd95dc1b698 Hostname: linux-uoch Coredump: /var/lib/systemd/coredump/core.vim.0.b5c251b86ab34674a2222cef102 Message: Process 2345 (vim) of user 0 dumped core. Stack trace of thread 2345: #0 0x00007f21dd87e2a7 kill (libc.so.6) #1 0x000000000050cb35 may_core_dump (vim) #2 0x00007f21ddbfec70 __restore_rt (libpthread.so.0) #3 0x00007f21dd92ea33 __select (libc.so.6) #4 0x000000000050b4e3 RealWaitForChar (vim) #5 0x000000000050b86b mch_inchar (vim) [...]
When you have multiple core dumps, coredumpctl info
displays all of them. Filter them by PID
,
COMM
(command), or EXE
(full path to
the executable), for example, all core dumps for Vim:
root #
coredumpctl info /bin/vim
See a single core dump by PID
:
root #
coredumpctl info 2345
Output the selected core to gdb
:
root #
coredumpctl gdb 2345
The asterisk in the PRESENT
column indicates that a
stored core dump is present. If the field is empty there is no stored core
dump, and coredumpctl
retrieves crash information from
the journal. You may control this behavior in
/etc/systemd/coredump.conf
with the
Storage
option:
Storage=none
, core dumps are logged in the journal, but not stored. This is useful to minimize collecting and storing sensitive information, for example for General Data Protection Regulation (GDPR) compliance.Storage=external
, cores are stored in/var/lib/systemd/coredump
Storage=journal
, cores are stored in thesystemd
journalStorage=both
, cores are stored in both locations
A new instance of systemd-coredump
is invoked for every core dump, so
configuration changes are applied with the next core dump, and there is no
need to restart any services.
Core dumps are not preserved after a system restart. You may save them
permanently with coredumpctl
. The following example
filters by the PID
and stores the core in
vim.dump
:
root #
coredumpctl -o vim.dump dump 2345
See man systemd-coredump
, man
coredumpctl
, and man coredump.conf
for complete
command and option listings.