Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
documentation.suse.com / Managing System Logs of SLE Micro

Managing System Logs of SLE Micro

Publication Date: 05 Dec 2024
WHAT?

System log file analysis is one of the most important tasks when analyzing the system. In fact, looking at the system log files should be the first thing to do when maintaining or troubleshooting a system. SUSE Linux Enterprise Micro automatically logs almost everything that happens on the system in detail.

WHY?

This article provides instructions on how to examine what happened on your system by viewing system logs.

EFFORT

It takes about 20 minutes to read and understand this article.

GOAL

You will get an overview where the log files reside and how to manage them.

REQUIREMENTS

  • root privileges.

1 Where to find system log files?

 

SLE Micro logs several types of messages, for example, from the kernel, SELinux or other services.

Kernel messages and messages of system services registered with systemd are logged in systemd journal (see Section 4, “The systemd logging system—journal”). Other system log files are located under the /var/log directory. SELinux messages are logged in /var/log/audit/audit.log. For details, refer to SELinux troubleshooting.

The following list provides an overview of all system log files found in SUSE Linux Enterprise Micro after a default installation. Depending on your installation scope, /var/log also contains log files from other services and applications not listed here. Some files and directories described below are “placeholders” and are only used when the corresponding application is installed. Most log files are only visible for the user root. Usually, you can view these files using an editor as they are in plain text.

Important
Important: Unsupported log files

utmp, wtmp and lastlog have been removed from SLE Micro and are no longed supported. If there are any applications that write into these log files, remember that the log files are incomplete then. wtmp has been replaced with wtmpdb, lastlog with lastlog2.

audit/

Logs from the audit framework.

ConsoleKit/

Logs of the ConsoleKit daemon (daemon for tracking what users are logged in and how they interact with the computer).

cups/

Access and error logs of the Common Unix Printing System (cups).

firewalld

Firewall logs.

krb5/

Log files from the Kerberos network authentication system.

chrony/

Logs from the Network Time Protocol daemon (chrony).

YaST2/

All YaST log files.

zypp/

libzypp log files. Refer to these files for the package installation history.

zypper.log

Logs from the command-line installer zypper.

2 Viewing and parsing /var/log files

 

You can view and parse plain text logs in /var/log using CLI commands as described further.

For viewing log files, use the commands less or more. Use head and tail to view the beginning or end of a log file. To view entries appended to a log file in real time, use tail -f. For information about how to use these tools, see their man pages.

To search for strings or regular expressions in log files, use grep. awk is useful for parsing and rewriting log files.

3 Managing log files with logrotate

 

Log files under /var/log grow daily and quickly become very large. logrotate is a tool that helps you manage log files and their growth. It allows automatic rotation, removal, compression, and mailing of log files. Log files can be handled periodically (daily, weekly, or monthly) or when exceeding a particular size.

logrotate is usually run daily by systemd, and thus usually modifies log files only once a day. However, exceptions occur when a log file is modified because of its size, if logrotate is run multiple times a day, or if --force is enabled. View the /var/lib/misc/logrotate.status file to find out when a particular file was last rotated.

logrotate can be configured to suit your needs. For details, refer to Section 3.1, “Configuring logrotate”.

3.1 Configuring logrotate

 

The main configuration file logrotate.conf defines, for example, how often to rotate logs or which tool is used for data compressions. Each service can have its own logrotate configuration in /etc/logrotate.d/.

3.1.1 Adjusting logrotate.conf

 

The default version of logrotate.conf resides in the /usr/etc/ directory. If the default does not suit your needs, copy the file to /etc/logrotate.conf and change the configuration values there. Do not modify the /usr/etc/ version as it may get overwritten with a system update. You can replace the following values:

weekly

The frequency of log rotation. You can use any of the values: hourly, daily, weekly, monthly or yearly.

maxage

You can specify a number of days that logs are kept.

rotate 4

The number determines the amount of log rotations to retain the rotated logs. When set to rotate 0, the logs are deleted immediately. When set to rotate -1, the logs are not removed until they reach the value of maxage.

dateext

If the option is present in the configuration file, rotated log file names receive a extension with a date in the format: logname.YYYYMMDD. If not present, the default file name scheme is: logname.1, logname.2.

compress

If commented out, the logs are not compressed.

compresscmd and uncompresscmd

Here, you can change the default compression and decompression tools by setting corresponding absolute paths to the tools.

include PATH

You can change the default location of the file with log rotation information. The default is /var/lib/misc/logrotate.status.

3.1.2 Service-specific logrotate configuration

 

Services and applications can have a specific logrotate configuration in /etc/logrotate.d. Besides the options mentioned in Section 3.1.1, “Adjusting logrotate.conf”, you can also use the following configurations:

missingok

The log rotation does not report errors if any of the specified log files are missing.

notifempty

An empty log file is not rotated.

delaycompress

The compression of rotated logs is postponed until the next log rotation.

sharedscripts

Denotes a section with scripts that should be executed only once regardless of the numbers of logs being rotated. If omitted, the scripts are executed for each log file being rotated.

size

Defines the size a log file can reach before the log rotation is initiated. This option may ignore the time scheduling. The value can be in megabytes (M), kilobytes (K) or bytes (B).

minsize

Logs are rotated according to the specified time schedule if their size exceeds this value. The value can be in megabytes (M), kilobytes (K) or bytes (B).

maxsize

Specifies the maximum log file size. Reaching this limit triggers rotation even when the time interval has not been reached. The value can be in megabytes (M), kilobytes (K) or bytes (B).

4 The systemd logging system—journal

 

systemd features its own logging system called journal. The journal itself is a system service managed by systemd—systemd-journald.service.

The service collects and stores logging data by maintaining structured indexed journals based on logging information received from the kernel, user processes, standard input, and system service errors. The systemd-journald service is enabled and started by default.

The journal stores log data in /var/log/journal/.

4.1 Usage of the journalctl command

 

This section introduces several common useful options to enhance the default journalctl behavior.

The journalctl command has the following syntax: 

    journalctl [options…] [matches…]
Tip
Tip: Messages related to a specific executable

To show all journal messages related to a specific executable, specify the full path to the executable: 

> sudo journalctl /usr/lib/systemd/systemd

The command takes the following options:

-f

Shows only the most recent journal messages, and prints new log entries as they are added to the journal.

-e

Prints the messages and jumps to the end of the journal, so that the latest entries are visible within the pager.

-r

Prints the messages of the journal in reverse order, so that the latest entries are listed first.

-k

Shows only kernel messages. This is equivalent to the field match _TRANSPORT=kernel.

-u

Shows only messages for the specified systemd unit. This is equivalent to the field match _SYSTEMD_UNIT=UNIT. 

> sudo journalctl -u apache2
 [...]
 Jun 03 10:07:11 pinkiepie systemd[1]: Starting The Apache Webserver...
 Jun 03 10:07:12 pinkiepie systemd[1]: Started The Apache Webserver.

4.2 Filtering journal logs

 

When journalctl is called without options, the command outputs the full content of the journal, the oldest entries listed first. The output can be filtered by specific options or journal fields.

4.2.1 Filtering based on time interval

 

You can filter the output of journalctl by specifying the starting and/or ending date. The date specification should be of the format 2014-06-30 9:17:16. If the time part is omitted, midnight is assumed. If seconds are omitted, :00 is assumed. If the date part is omitted, the current day is assumed. Instead of numeric expression, you can specify the keywords “yesterday,” “today” or “tomorrow.” They refer to midnight of the day before the current day, of the current day, or of the day after the current day. If you specify “now,” it refers to the current time. You can also specify relative times prefixed with - or +, referring to times before or after the current time.

Show only new messages since now, and update the output continuously: 

> sudo journalctl --since "now" -f

Display all messages from midnight until 3:20 a.m. 

> sudo journalctl --since "today" --until "3:20"

4.2.2 Filtering based on a boot number

 

journalctl can filter messages based on a specific system boot. To list all available boots, run 

> sudo journalctl --list-boots
 -1 097ed2cd99124a2391d2cffab1b566f0 Mon 2014-05-26 08:36:56 EDT—Fri 2014-05-30 05:33:44 EDT
  0 156019a44a774a0bb0148a92df4af81b Fri 2014-05-30 05:34:09 EDT—Fri 2014-05-30 06:15:01 EDT

The first column lists the boot offset: 0 for the current boot, -1 for the previous one, -2 for the one prior to that, etc. The second column contains the boot ID, followed by the limiting time stamps of the specific boot.

Show all messages from the current boot: 

> sudo journalctl -b

If you need to see journal messages from the previous boot, add an offset parameter. The following example outputs the previous boot messages: 

> sudo journalctl -b -1

Another way is to list boot messages based on the boot ID. For this purpose, use the _BOOT_ID field: 

> sudo journalctl _BOOT_ID=156019a44a774a0bb0148a92df4af81b

4.2.3 Filtering based on fields

 

You can filter the output of the journal by specific fields. The syntax of a field to be matched is FIELD_NAME=MATCHED_VALUE, such as _SYSTEMD_UNIT=httpd.service. You can specify multiple matches in a single query to filter the output messages even more. See man 7 systemd.journal-fields for a list of default fields.

Show messages produced by a specific process ID: 

> sudo journalctl _PID=1039

Show messages belonging to a specific user ID: 

# journalctl _UID=1000

Show messages from the kernel ring buffer (the same as dmesg produces): 

> sudo journalctl _TRANSPORT=kernel

Show messages from the service's standard or error output: 

> sudo journalctl _TRANSPORT=stdout

Show messages produced by a specified service only: 

> sudo journalctl _SYSTEMD_UNIT=avahi-daemon.service

If two different fields are specified, only entries that match both expressions at the same time are shown: 

> sudo journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=1488

If two matches refer to the same field, all entries matching either expression are shown: 

> sudo journalctl _SYSTEMD_UNIT=avahi-daemon.service _SYSTEMD_UNIT=dbus.service

You can use the + separator to combine two expressions in a logical OR. The following example shows all messages from the Avahi service process with the process ID 1480 together with all messages from the D-Bus service: 

> sudo journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=1480 + _SYSTEMD_UNIT=dbus.service

4.3 Journald configuration

 
> sudo systemctl restart systemd-journald

4.3.1 Changing the journal size limit

 

The journal log data uses up to 10% of the file system on which /var/log/journal resides. For example, if /var/log/journal is located on a 30 GB /var partition, the journal may use up to 3 GB of the disk space. To change this limit, change (and uncomment) the SystemMaxUse option: 

SystemMaxUse=50M

4.3.2 Forwarding the journal to /dev/ttyX

 

You can forward the journal to a terminal device to inform you about system messages on a preferred terminal screen, for example, /dev/tty12. Change the following journald options to 

ForwardToConsole=yes
TTYPath=/dev/tty12