Managing System Logs of SUSE Linux Micro
- WHAT?
System log file analysis is one of the most important tasks when analyzing the system. In fact, looking at the system log files should be the first thing to do when maintaining or troubleshooting a system. SUSE Linux Micro automatically logs almost everything that happens on the system in detail.
- WHY?
This article provides instructions on how to examine what happened on your system by viewing system logs.
- EFFORT
It takes about 20 minutes to read and understand this article.
- GOAL
You will get an overview where the log files reside and how to manage them.
- REQUIREMENTS
root
privileges.
1 Where to find system log files? #
SUSE Linux Micro logs several types of messages, for example, from the kernel, SELinux or other services.
Kernel messages and messages of system services registered with systemd
are logged in systemd
journal (see
Section 4, “The systemd
logging system—journal”). Other system log files are
located under the /var/log
directory. SELinux messages
are logged in /var/log/audit/audit.log
. For details,
refer to
SELinux
troubleshooting.
The following list provides an overview of all system log files found in
SUSE Linux Micro after a default installation. Depending on your installation
scope, /var/log
also contains log files from other
services and applications not listed here. Some files and directories
described below are “placeholders” and are only used when the
corresponding application is installed. Most log files are only visible for
the user root
. Usually, you can view these files using an editor as
they are in plain text.
utmp
, wtmp
and
lastlog
have been removed from SUSE Linux Micro and
are no longed supported. If there are any applications that write into
these log files, remember that the log files are incomplete then.
wtmp
has been replaced with wtmpdb
,
lastlog
with lastlog2
.
audit/
Logs from the audit framework.
ConsoleKit/
Logs of the
ConsoleKit
daemon (daemon for tracking what users are logged in and how they interact with the computer).cups/
Access and error logs of the Common Unix Printing System (
cups
).firewalld
Firewall logs.
krb5/
Log files from the Kerberos network authentication system.
chrony/
Logs from the Network Time Protocol daemon (
chrony
).YaST2/
All YaST log files.
zypp/
libzypp
log files. Refer to these files for the package installation history.zypper.log
Logs from the command-line installer
zypper
.
2 Viewing and parsing /var/log
files #
You can view and parse plain text logs in /var/log
using CLI commands as described further.
For viewing log files, use the commands less
or
more
. Use head
and
tail
to view the beginning or end of a log file. To view
entries appended to a log file in real time, use tail
-f
. For information about how to use these tools, see
their man pages.
To search for strings or regular expressions in log files, use
grep
. awk
is useful for parsing and
rewriting log files.
3 Managing log files with logrotate
#
Log files under /var/log
grow daily and quickly
become very large. logrotate
is a tool that helps
you manage log files and their growth. It allows automatic rotation,
removal, compression, and mailing of log files. Log files can be
handled periodically (daily, weekly, or monthly) or when exceeding a
particular size.
logrotate
is usually run daily by systemd
, and thus
usually modifies log files only once a day. However, exceptions occur when
a log file is modified because of its size, if logrotate
is run multiple times a day, or if --force
is enabled.
View the /var/lib/misc/logrotate.status
file to find
out when a particular file was last rotated.
logrotate
can be configured to suit your needs. For
details, refer to Section 3.1, “Configuring logrotate
”.
3.1 Configuring logrotate
#
The main configuration file logrotate.conf
defines, for example, how often to rotate logs or which tool is used
for data compressions. Each service can have its own logrotate
configuration in /etc/logrotate.d/
.
3.1.1 Adjusting logrotate.conf
#
The default version of logrotate.conf
resides in the
/usr/etc/
directory. If the default does not suit
your needs, copy the file to /etc/logrotate.conf
and
change the configuration values there. Do not modify the
/usr/etc/
version as it may get overwritten with a
system update. You can replace the following values:
weekly
The frequency of log rotation. You can use any of the values:
hourly
,daily
,weekly
,monthly
oryearly
.maxage
You can specify a number of days that logs are kept.
rotate 4
The number determines the amount of log rotations to retain the rotated logs. When set to
rotate 0
, the logs are deleted immediately. When set torotate -1
, the logs are not removed until they reach the value ofmaxage
.dateext
If the option is present in the configuration file, rotated log file names receive a extension with a date in the format:
logname.YYYYMMDD
. If not present, the default file name scheme is:logname.1
,logname.2
.compress
If commented out, the logs are not compressed.
compresscmd
anduncompresscmd
Here, you can change the default compression and decompression tools by setting corresponding absolute paths to the tools.
include PATH
You can change the default location of the file with log rotation information. The default is
/var/lib/misc/logrotate.status
.
3.1.2 Service-specific logrotate configuration #
Services and applications can have a specific logrotate configuration in
/etc/logrotate.d
. Besides the options mentioned in
Section 3.1.1, “Adjusting logrotate.conf
”, you
can also use the following configurations:
missingok
The log rotation does not report errors if any of the specified log files are missing.
notifempty
An empty log file is not rotated.
delaycompress
The compression of rotated logs is postponed until the next log rotation.
sharedscripts
Denotes a section with scripts that should be executed only once regardless of the numbers of logs being rotated. If omitted, the scripts are executed for each log file being rotated.
size
Defines the size a log file can reach before the log rotation is initiated. This option may ignore the time scheduling. The value can be in megabytes (M), kilobytes (K) or bytes (B).
minsize
Logs are rotated according to the specified time schedule if their size exceeds this value. The value can be in megabytes (M), kilobytes (K) or bytes (B).
maxsize
Specifies the maximum log file size. Reaching this limit triggers rotation even when the time interval has not been reached. The value can be in megabytes (M), kilobytes (K) or bytes (B).
4 The systemd
logging system—journal #
systemd
features its own logging system called
journal. The journal itself is a system service
managed by systemd
—systemd-journald.service
.
The service collects and stores logging data by maintaining structured
indexed journals based on logging information received from the kernel,
user processes, standard input, and system service errors. The
systemd-journald
service is enabled and started by
default.
The journal stores log data in /var/log/journal/
.
4.1 Usage of the journalctl
command #
This section introduces several common useful options to enhance the
default journalctl
behavior.
The journalctl
command has the following syntax:
journalctl [options…] [matches…]
To show all journal messages related to a specific executable, specify the full path to the executable:
>
sudo
journalctl /usr/lib/systemd/systemd
The command takes the following options:
- -f
Shows only the most recent journal messages, and prints new log entries as they are added to the journal.
- -e
Prints the messages and jumps to the end of the journal, so that the latest entries are visible within the pager.
- -r
Prints the messages of the journal in reverse order, so that the latest entries are listed first.
- -k
Shows only kernel messages. This is equivalent to the field match
_TRANSPORT=kernel
.- -u
Shows only messages for the specified
systemd
unit. This is equivalent to the field match_SYSTEMD_UNIT=UNIT
.>
sudo
journalctl -u apache2 [...] Jun 03 10:07:11 pinkiepie systemd[1]: Starting The Apache Webserver... Jun 03 10:07:12 pinkiepie systemd[1]: Started The Apache Webserver.
4.2 Filtering journal logs #
When journalctl
is called without options, the
command outputs the full content of the journal, the oldest entries
listed first. The output can be filtered by specific options or journal
fields.
4.2.1 Filtering based on time interval #
You can filter the output of journalctl
by specifying
the starting and/or ending date. The date specification should be of the
format 2014-06-30 9:17:16
. If the time part is
omitted, midnight is assumed. If seconds are omitted,
:00
is assumed. If the date part is omitted, the
current day is assumed. Instead of numeric expression, you can specify
the keywords “yesterday,” “today” or
“tomorrow.” They refer to midnight of the day before the
current day, of the current day, or of the day after the current day. If
you specify “now,” it refers to the current time. You can
also specify relative times prefixed with -
or
+
, referring to times before or after the current
time.
Show only new messages since now, and update the output continuously:
>
sudo
journalctl --since "now" -f
Display all messages from midnight until 3:20 a.m.
>
sudo
journalctl --since "today" --until "3:20"
4.2.2 Filtering based on a boot number #
journalctl
can filter messages based on a specific
system boot. To list all available boots, run
>
sudo
journalctl --list-boots -1 097ed2cd99124a2391d2cffab1b566f0 Mon 2014-05-26 08:36:56 EDT—Fri 2014-05-30 05:33:44 EDT 0 156019a44a774a0bb0148a92df4af81b Fri 2014-05-30 05:34:09 EDT—Fri 2014-05-30 06:15:01 EDT
The first column lists the boot offset: 0
for the
current boot, -1
for the previous one,
-2
for the one prior to that, etc. The second column
contains the boot ID, followed by the limiting time stamps of the
specific boot.
Show all messages from the current boot:
>
sudo
journalctl -b
If you need to see journal messages from the previous boot, add an offset parameter. The following example outputs the previous boot messages:
>
sudo
journalctl -b -1
Another way is to list boot messages based on the boot ID. For this purpose, use the _BOOT_ID field:
>
sudo
journalctl _BOOT_ID=156019a44a774a0bb0148a92df4af81b
4.2.3 Filtering based on fields #
You can filter the output of the journal by specific fields. The syntax
of a field to be matched is FIELD_NAME=MATCHED_VALUE
,
such as _SYSTEMD_UNIT=httpd.service
. You can specify
multiple matches in a single query to filter the output messages even
more. See man 7 systemd.journal-fields
for a list of
default fields.
Show messages produced by a specific process ID:
>
sudo
journalctl _PID=1039
Show messages belonging to a specific user ID:
# journalctl _UID=1000
Show messages from the kernel ring buffer (the same as
dmesg
produces):
>
sudo
journalctl _TRANSPORT=kernel
Show messages from the service's standard or error output:
>
sudo
journalctl _TRANSPORT=stdout
Show messages produced by a specified service only:
>
sudo
journalctl _SYSTEMD_UNIT=avahi-daemon.service
If two different fields are specified, only entries that match both expressions at the same time are shown:
>
sudo
journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=1488
If two matches refer to the same field, all entries matching either expression are shown:
>
sudo
journalctl _SYSTEMD_UNIT=avahi-daemon.service _SYSTEMD_UNIT=dbus.service
You can use the +
separator to combine two expressions
in a logical OR
. The following example shows all
messages from the Avahi service process with the process ID 1480 together
with all messages from the D-Bus service:
>
sudo
journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=1480 + _SYSTEMD_UNIT=dbus.service
4.3 Journald configuration #
>
sudo
systemctl restart systemd-journald
4.3.1 Changing the journal size limit #
The journal log data uses up to 10% of the file system on which
/var/log/journal
resides. For example, if
/var/log/journal
is located on a 30 GB
/var
partition, the journal may use up to 3 GB
of the disk space. To change this limit, change (and uncomment) the
SystemMaxUse
option:
SystemMaxUse=50M
4.3.2 Forwarding the journal to /dev/ttyX
#
You can forward the journal to a terminal device to inform you about
system messages on a preferred terminal screen, for example,
/dev/tty12
. Change the following journald options to
ForwardToConsole=yes TTYPath=/dev/tty12
5 Legal Notice #
Copyright© 2006–2024 SUSE LLC and contributors. All rights reserved.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”.
For SUSE trademarks, see https://www.suse.com/company/legal/. All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its affiliates. Asterisks (*) denote third-party trademarks.
All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors, nor the translators shall be held liable for possible errors or the consequences thereof.