3 System log files #
System log file analysis is one of the most important tasks when analyzing
the system. In fact, looking at the system log files should be the first
thing to do when maintaining or troubleshooting a system. SUSE Linux Enterprise Server
automatically logs almost everything that happens on the system in detail.
Since the move to systemd
, kernel messages and messages of system
services registered with systemd
are logged in systemd
journal
(see Chapter 21, journalctl
: query the systemd
journal). Other log files (mainly those of
system applications) are written in plain text and can be easily read
using an editor or pager. It is also possible to parse them using scripts.
This allows you to filter their content.
3.1 System log files in /var/log/
#
System log files are always located under the
/var/log
directory. The following list presents an
overview of all system log files from SUSE Linux Enterprise Server present after a
default installation. Depending on your installation scope,
/var/log
also contains log files from other services
and applications not listed here. Some files and directories described
below are “placeholders” and are only used, when the
corresponding application is installed. Most log files are only visible
for the user root
.
apparmor/
AppArmor log files. For more information about AppArmor, see Part V, “Confining privileges with AppArmor”.
audit/
Logs from the audit framework. See Part VII, “The Linux Audit Framework” for details.
ConsoleKit/
Logs of the
ConsoleKit
daemon (daemon for tracking what users are logged in and how they interact with the computer).cups/
Access and error logs of the Common Unix Printing System (
cups
).firewall
Firewall logs.
gdm/
Log files from the GNOME display manager.
krb5/
Log files from the Kerberos network authentication system.
lastlog
A database containing information on the last login of each user. Use the command
lastlog
to view. Seeman 8 lastlog
for more information.localmessages
Log messages of certain boot scripts, for example, the log of the DHCP client.
mail*
Mail server (
postfix
,sendmail
) logs.messages
This is the default place where all kernel and system log messages go and should be the first place (along with
/var/log/warn
) to look at in case of problems.NetworkManager
NetworkManager log files.
news/
Log messages from a news server.
chrony/
Logs from the Network Time Protocol daemon (
chrony
).pk_backend_zypp*
PackageKit (with
libzypp
back-end) log files.samba/
Log files from Samba, the Windows SMB/CIFS file server.
warn
Log of all system warnings and errors. This should be the first place (along with the output of the
systemd
journal) to look in case of problems.wtmp
Database of all login/logout activities, and remote connections. Use the command
last
to view. Seeman 1 last
for more information.Xorg.NUMBER.log
X.Org start-up log file. Refer to these files in case you have problems starting X.Org.
The NUMBER in the file name is the display number. For example, the default
Xorg.0.log
is the log for display number0
, andXorg.1.log
is the log for display number1
. Copies from previous X.Org starts are named asXorg.NUMBER.log.old
.NoteThe X.Org log files are available in the
/var/log/
directory only if you start an X.Org session asroot
. If you start an X.Org session as any other user, you can locate the log files in the~/.local/share/xorg/
directory.YaST2/
All YaST log files.
zypp/
libzypp
log files. Refer to these files for the package installation history.zypper.log
Logs from the command line installer
zypper
.
3.2 Viewing and parsing log files #
To view log files, you can use any text editor. There is also a simple YaST module for viewing the system log available in the YaST control center under
› .
For viewing log files in a text console, use the commands
less
or more
. Use
head
and tail
to view the beginning
or end of a log file. To view entries appended to a log file in real-time
use tail
-f
. For information about
how to use these tools, see their man pages.
To search for strings or regular expressions in log files use
grep
. awk
is useful for parsing and
rewriting log files.
3.3 Managing log files with logrotate
#
Log files under /var/log
grow on a daily basis and
quickly become large. logrotate
is a tool that
helps you manage log files and their growth. It allows automatic
rotation, removal, compression, and mailing of log files. Log files can
be handled periodically (daily, weekly, or monthly) or when exceeding a
particular size.
logrotate
is run daily by systemd
,
and thus modifies log files only once a day. However, exceptions
occur when a log file is modified because of its size, if
logrotate
is run multiple times a day, or if
--force
is enabled. Use
/var/lib/misc/logrotate.status
to find out when a
particular file was last rotated.
The main configuration file of logrotate
is
/etc/logrotate.conf
. System packages and
programs that produce log files (for example,
apache2
) put their own
configuration files in the /etc/logrotate.d/
directory. The content of /etc/logrotate.d/
is
included via /etc/logrotate.conf
.
/etc/logrotate.conf
#
# see "man logrotate" for details # rotate log files weekly weekly # keep 4 weeks worth of backlogs rotate 4 # create new (empty) log files after rotating old ones create # use date as a suffix of the rotated file dateext # uncomment this if you want your log files compressed #compress # comment these to switch compression to use gzip or another # compression scheme compresscmd /usr/bin/bzip2 uncompresscmd /usr/bin/bunzip2 # RPM packages drop log rotation information into this directory include /etc/logrotate.d
The create
option pays heed to the modes and
ownership of files specified in /etc/permissions*
.
If you modify these settings, make sure no conflicts arise.
3.4 Monitoring log files with logwatch
#
logwatch
is a customizable, pluggable log-monitoring
script. It parses system logs, extracts the important information and
presents them in a human readable manner. To use
logwatch
, install the logwatch package.
logwatch
can either be used at the command line to
generate on-the-fly reports, or via cron
to regularly create custom
reports. Reports can either be printed on the screen, saved to a file, or
be mailed to a specified address. The latter is especially useful when
automatically generating reports via cron
.
On the command line, you can tell logwatch
for which
service and time span to generate a report and how much detail should be
included:
# Detailed report on all kernel messages from yesterday logwatch --service kernel --detail High --range Yesterday --print # Low detail report on all sshd events recorded (incl. archived logs) logwatch --service sshd --detail Low --range All --archives --print # Mail a report on all smartd messages from May 5th to May 7th to root@localhost logwatch --service smartd --range 'between 5/5/2005 and 5/7/2005' \ --mailto root@localhost --print
The --range
option has got a complex syntax—see
logwatch
--range help
for details. A
list of all services that can be queried is available with the following
command:
>
ls /usr/share/logwatch/default.conf/services/ | sed 's/\.conf//g'
logwatch
can be customized to great detail. However,
the default configuration should be sufficient. The default
configuration files are located under
/usr/share/logwatch/default.conf/
. Never change them
because they would get overwritten again with the next update. Rather
place custom configuration in /etc/logwatch/conf/
(you may use the default configuration file as a template, though). A
detailed HOWTO on customizing logwatch
is available at
/usr/share/doc/packages/logwatch/HOWTO-Customize-LogWatch
.
The following configuration files exist:
logwatch.conf
The main configuration file. The default version is extensively commented. Each configuration option can be overwritten on the command line.
ignore.conf
Filter for all lines that should globally be ignored by
logwatch
.services/*.conf
The service directory holds configuration files for each service you can generate a report for.
logfiles/*.conf
Specifications on which log files should be parsed for each service.
3.5 Configuring mail forwarding for root
#
System daemons, cron
jobs, systemd
timers, and other applications can generate messages and send them to the
root
user of the system. By default, each user account owns a local
mailbox and will be notified about new mail messages upon login.
These messages can contain security relevant reports and incidents that require a quick response by the system administrator. To get notified about these messages in a timely fashion, it is strongly recommended to forward these mails to a dedicated remote e-mail account that is regularly checked.
root
user #
To forward mail for the root
user, perform the following steps:
Install the yast2-mail package:
#
zypper in yast2-mail
Run the interactive YaST mail configuration:
#
yast mail
Choose
as and proceed with .Enter the address of the
. If necessary, configure . It is strongly recommended to to prevent potentially sensitive system data from being sent unencrypted over the network. Proceed with .Enter the e-mail address to
and the configuration.Important: Do not accept remote SMTP connectionsDo not enable , otherwise the local machine will act as a mail relay.
Send a message to test whether mail forwarding works correctly:
>
mail root
subject: test test .Use the
mailq
command to verify that the test message has been sent. Upon success, the queue should be empty. The message should be received by the dedicated mail address configured previously.
Depending on the number of managed machines and the number of persons who need to be informed about system events, different e-mail address models can be established:
Collect messages from different systems in an e-mail account that is only accessed by a single person.
Collect messages from different systems in a group e-mail account (aliases or mailing list) that can be accessed by all relevant persons.
Create separate e-mail accounts for each system.
It is crucial that administrators regularly check the related e-mail accounts. To facilitate this effort and identify important events, avoid sending unnecessary information. Configure applications to only send relevant information.
3.6 Forwarding log messages to a central syslog server #
System log data can be forwarded from individual systems to a central syslog server on the network. This allows administrators to get an overview of events on all hosts, and prevents attackers that succeed in taking over a system from manipulating system logs to cover their tracks.
Setting up a central syslog server consists of two parts. First you configure the central log server, then the clients for remote logging.
3.6.1 Set up the central syslog server #
rsyslog
server #To set up a central syslog server, perform the following steps:
Edit the configuration file
/etc/rsyslog.d/remote.conf
.Uncomment the following lines in the
UDP Syslog Server
orTCP Syslog Server
section of the configuration file. Assign an IP address and port forrsyslogd
.TCP example:
$ModLoad imtcp.so $UDPServerAddress IP1 $InputTCPServerRun PORT2
UDP example:
IP address of the interface for
rsyslogd
to listen on. If no address is given, the daemon listens on all interfaces.Port for
rsyslogd
to listen on. Select a privileged port below 1024. The default is 514.Important: TCP versus UDP protocolTraditionally syslog uses the UDP protocol to transmit log messages over the network. This involves less overhead, but lacks reliability. Log messages can get lost under high load.
The TCP protocol is more reliable and should be preferred over UDP.
Note:UDPServerAddress
with TCPThe
$UDPServerAddress
configuration parameter in the TCP example is no error. Despite its name it is used for both TCP and UDP.Save the file.
Restart the
rsyslog
service:>
sudo
systemctl restart rsyslog.service
Open the respective port in the firewall. For
firewalld
with TCP on port 514 run:>
sudo
firewall-cmd --add-port 514/tcp --permanent
>
sudo
firewall-cmd --reload
You have now configured the central syslog server. Next, configure clients for remote logging.
3.6.2 Set up the client machines #
To configure a machine for remote logging to a central syslog server, perform the following steps:
Edit the configuration file
/etc/rsyslog.d/remote.conf
.Uncomment the appropriate line (TCP or UDP) and replace
remote-host
with the address of the central syslog server set up in Section 3.6.1, “Set up the central syslog server”.TCP example:
# Remote Logging using TCP for reliable delivery # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional *.* @@remote-host
UDP example:
# Remote Logging using UDP # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional *.* @remote-host
Save the file.
Restart the
rsyslog
service:>
sudo
systemctl restart rsyslog.service
Verify the proper function of the syslog forwarding:
>
logger "hello world"
The log message
hello world
should now appear on the central syslog server.
You have now configured a system for remote logging to your central syslog server. Repeat this procedure for all systems that should log remotely.
3.6.3 More information #
This basic setup does not include encryption and is only suitable for trusted internal networks. TLS encryption is strongly recommended, but requires a certificate infrastructure.
In this configuration, all messages from remote hosts are treated the same on the central syslog server. Consider filtering messages into separate files by remote host or classify them by message category.
For more information about encryption, filtering, and other advanced topics, consult the RSyslog documentation at https://www.rsyslog.com/doc/master/index.html#manual.
3.7 Using logger
to make system log entries #
logger
is a tool for making entries in the system log.
It provides a shell command interface to the rsyslogd system log module.
For example, the following line outputs its message in
/var/log/messages
or directly in the journal (if no
logging facility is running):
>
logger -t Test "This message comes from $USER"
Depending on the current user and host name, the log contains a line similar to this:
Sep 28 13:09:31 venus Test: This message comes from tux