14 The systemd daemon #

systemd is a system and session manager for Linux, compatible with System V and LSB init scripts. The main features are:

A unit configuration file contains information about a service, a socket, a device, a mount point, an automount point, a swap file or partition, a start-up target, a watched file system path, a timer controlled and supervised by systemd, a temporary system state snapshot, a resource management slice or a group of externally created processes.

“Unit file” is a generic term used by systemd for the following:

For more information about systemd unit files, see http://www.freedesktop.org/software/systemd/man/systemd.unit.html

Subcommands for managing services are the same as for managing a service with System V init (start, stop, ...). The general syntax for service management commands is as follows:

systemd allows you to manage several services in one go. Instead of executing init scripts one after the other as with System V init, execute a command like the following:

systemctl start MY_1ST_SERVICE MY_2ND_SERVICE

To list all services available on the system:

systemctl list-unit-files --type=service

The following table lists the most important service management commands for systemd and System V init:

start

start

stop

stop

restart

restart

try-restart

try-restart

reload

reload

reload-or-restart

n/a

reload-or-try-restart

n/a

status

status

is-active

status

The service management commands mentioned in the previous section let you manipulate services for the current session. systemd also lets you permanently enable or disable services, so they are automatically started when requested or are always unavailable. You can either do this by using YaST, or on the command line.

14.2.2.1 Enabling/disabling services on the command line #

 

The following table lists enabling and disabling commands for systemd and System V init:

Important: Service start

When enabling a service on the command line, it is not started automatically. It is scheduled to be started with the next system start-up or runlevel/target change. To immediately start a service after having enabled it, explicitly run systemctl start MY_SERVICE or rc MY_SERVICE start.

Table 14.2: Commands for enabling and disabling services #

 

Task	systemd Command	System V init Command
Enabling.	systemctl enable MY_SERVICE(S)	insserv MY_SERVICE(S), chkconfig -a MY_SERVICE(S)
Disabling.	systemctl disable MY_SERVICE(S).service	insserv -r MY_SERVICE(S), chkconfig -d MY_SERVICE(S)
Checking.  Shows whether a service is enabled or not.	systemctl is-enabled MY_SERVICE	chkconfig MY_SERVICE
Re-enabling.  Similar to restarting a service, this command first disables and then enables a service. Useful to re-enable a service with its defaults.	systemctl reenable MY_SERVICE	n/a
Masking.  After “disabling” a service, it can still be started manually. To disable a service, you need to mask it. Use with care.	systemctl mask MY_SERVICE	n/a
Unmasking.  A service that has been masked can only be used again after it has been unmasked.	systemctl unmask MY_SERVICE	n/a

With System V init the system was booted into a so-called “Runlevel”. A runlevel defines how the system is started and what services are available in the running system. Runlevels are numbered; the most commonly known ones are 0 (shutting down the system), 3 (multiuser with network) and 5 (multiuser with network and display manager).

systemd introduces a new concept by using so-called “target units”. However, it remains fully compatible with the runlevel concept. Target units are named rather than numbered and serve specific purposes. For example, the targets local-fs.target and swap.target mount local file systems and swap spaces.

The target graphical.target provides a multiuser system with network and display manager capabilities and is equivalent to runlevel 5. Complex targets, such as graphical.target act as “meta” targets by combining a subset of other targets. Since systemd makes it easy to create custom targets by combining existing targets, it offers great flexibility.

The following list shows the most important systemd target units. For a full list refer to man 7 systemd.special.

To remain compatible with the System V init runlevel system, systemd provides special targets named runlevelX.target mapping the corresponding runlevels numbered X.

To inspect the current target, use the command: systemctl get-default

Important: systemd ignores /etc/inittab

The runlevels in a System V init system are configured in /etc/inittab. systemd does not use this configuration. Refer to Section 14.5.5, “Creating Custom Targets” for instructions on how to create your own bootable target.

systemd offers the means to analyze the system start-up process. You can review the list of all services and their status (rather than having to parse /varlog/). systemd also allows you to scan the start-up procedure to find out how much time each service start-up consumes.

root # systemctl
UNIT                        LOAD   ACTIVE SUB       JOB DESCRIPTION
[...]
iscsi.service               loaded active exited    Login and scanning of iSC+
kmod-static-nodes.service   loaded active exited    Create list of required s+
libvirtd.service            loaded active running   Virtualization daemon
nscd.service                loaded active running   Name Service Cache Daemon
ntpd.service                loaded active running   NTP Server Daemon
polkit.service              loaded active running   Authorization Manager
postfix.service             loaded active running   Postfix Mail Transport Ag+
rc-local.service            loaded active exited    /etc/init.d/boot.local Co+
rsyslog.service             loaded active running   System Logging Service
[...]
LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

161 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

root # systemctl --failed
UNIT                   LOAD   ACTIVE SUB    JOB DESCRIPTION
apache2.service        loaded failed failed     apache
NetworkManager.service loaded failed failed     Network Manager
plymouth-start.service loaded failed failed     Show Plymouth Boot Screen

[...]

systemd.log_level=debug systemd.log_target=kmsg

dmesg -T | less

systemd is compatible with System V, allowing you to still use existing System V init scripts. However, there is at least one known issue where a System V init script does not work with systemd out of the box: starting a service as a different user via su or sudo in init scripts will result in a failure of the script, producing an “Access denied” error.

When changing the user with su or sudo, a PAM session is started. This session will be terminated after the init script is finished. As a consequence, the service that has been started by the init script will also be terminated. To work around this error, proceed as follows:

systemd unit files shipped by SUSE are stored in /usr/lib/systemd/. Customized unit files and unit file drop-ins are stored in /etc/systemd/.

Warning: Preventing your customization from being overwritten

When customizing systemd, always use the directory /etc/systemd/ instead of /usr/lib/systemd/. Otherwise your changes will be overwritten by the next update of systemd.

Drop-in files (or drop-ins) are partial unit files that override only specific settings of the unit file. Drop-ins have higher precedence over main configuration files. The command systemctl edit SERVICE starts the default text editor and creates a directory with an empty override.conf file in /etc/systemd/system/NAME.service.d/. The command also ensures that the running systemd process is notified about the changes.

For example, to change the amount of time that the system waits for MariaDB to start, run sudo systemctl edit mariadb.service and edit the opened file to include the modified lines only:

# Configures the time to wait for start-up/stop
TimeoutSec=300

Adjust the TimeoutSec value and save the changes. To enable the changes, run sudo systemctl daemon-reload.

For further information, refer to the man pages that can be evoked with the man 1 systemctl command.

Warning: Creating a copy of a full unit file

If you use the --full option in the systemctl edit --full SERVICE command, a copy of the original unit file is created where you can modify specific options. We do not recommend such customization because when the unit file is updated by SUSE, its changes are overridden by the customized copy in the /etc/systemd/system/ directory. Moreover, if SUSE provides updates to distribution drop-ins, they will override the copy of the unit file created with --full. To prevent this confusion and always have your customization valid, use drop-ins.

Apart from using the systemctl edit command, you can create drop-ins manually to have more control over their priority. Such drop-ins let you extend both unit and daemon configuration files without having to edit or override the files themselves. They are stored in the following directories:

Tip

See the man page man 5 systemd.unit for the full list of unit search paths.

For example, to disable the rate limiting that is enforced by the default setting of systemd-journald, follow these steps:

Note: Avoiding name conflicts

To avoid name conflicts between your drop-ins and files shipped by SUSE, it is recommended to prefix all drop-ins with a two-digit number and a dash, for example, 80-override.conf.

The following ranges are reserved:

• 0-19 is reserved for systemd upstream.

• 20-29 is reserved for systemd shipped by SUSE.

• 30-39 is reserved for SUSE packages other than systemd.

• 40-49 is reserved for third party packages.

• 50 is reserved for unit drop-in files created with systemctl set-property.

Use a two-digit number above this range to ensure that none of the drop-ins shipped by SUSE can override your own drop-ins.

Tip

You can use systemctl cat $UNIT to list and verify which files are taken into account in the units configuration.

Tip

Because the configuration of systemd components can be scattered across different places on the file system, it might be hard to get a global overview. To inspect the configuration of a systemd component, use the following commands:

• systemctl cat UNIT_PATTERN prints configuration files related to one or more systemd units, for example:

  tux > systemctl cat atd.service

• systemd-analyze cat-config DAEMON_NAME_OR_PATH copies the contents of a configuration file and drop-ins for a systemd daemon, for example:

  tux > systemd-analyze cat-config systemd/journald.conf

Since the release of SUSE Linux Enterprise Server 15, the xinetd infrastructure has been removed. This section outlines how to convert existing custom xinetd service files to systemd sockets.

For each xinetd service file, you need at least two systemd unit files: the socket file (*.socket) and an associated service file (*.service). The socket file tells systemd which socket to create, and the service file tells systemd which executable to start.

Consider the following example xinetd service file:

root # cat /etc/xinetd.d/example
service example
{
  socket_type = stream
  protocol = tcp
  port = 10085
  wait = no
  user = user
  group = users
  groups = yes
  server = /usr/libexec/example/exampled
  server_args = -auth=bsdtcp exampledump
  disable = no
}

To convert it to systemd, you need the following two matching files:

root # cat /usr/lib/systemd/system/example.socket
[Socket]
ListenStream=0.0.0.0:10085
Accept=false

[Install]
WantedBy=sockets.target

root # cat /usr/lib/systemd/system/example.service
[Unit]
Description=example

[Service]
ExecStart=/usr/libexec/example/exampled -auth=bsdtcp exampledump
User=user
Group=users
StandardInput=socket

For a complete list of the systemd “socket” and “service” file options, refer to the systemd.socket and systemd.service manual pages (man 5 systemd.socket, man 5 systemd.service).

On System V init SUSE systems, runlevel 4 is unused to allow administrators to create their own runlevel configuration. systemd allows you to create any number of custom targets. It is suggested to start by adapting an existing target such as graphical.target.

systemd supports cleaning temporary directories regularly. The configuration from the previous system version is automatically migrated and active. tmpfiles.d—which is responsible for managing temporary files—reads its configuration from /etc/tmpfiles.d/*.conf , /run/tmpfiles.d/*.conf, and /usr/lib/tmpfiles.d/*.conf files. Configuration placed in /etc/tmpfiles.d/*.conf overrides related configurations from the other two directories (/usr/lib/tmpfiles.d/*.conf is where packages store their configuration files).

The configuration format is one line per path containing action and path, and optionally mode, ownership, age and argument fields, depending on the action. The following example unlinks the X11 lock files:

Type Path               Mode UID  GID  Age Argument
r    /tmp/.X[0-9]*-lock

To get the status the tmpfile timer:

systemctl status systemd-tmpfiles-clean.timer
systemd-tmpfiles-clean.timer - Daily Cleanup of Temporary Directories
 Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.timer; static)
 Active: active (waiting) since Tue 2014-09-09 15:30:36 CEST; 1 weeks 6 days ago
   Docs: man:tmpfiles.d(5)
         man:systemd-tmpfiles(8)

Sep 09 15:30:36 jupiter systemd[1]: Starting Daily Cleanup of Temporary Directories.
Sep 09 15:30:36 jupiter systemd[1]: Started Daily Cleanup of Temporary Directories.

For more information on temporary files handling, see man 5 tmpfiles.d.

Section 14.6.9, “Debugging services” explains how to view log messages for a given service. However, displaying log messages is not restricted to service logs. You can also access and query the complete log messages written by systemd—the so-called “Journal”. Use the command journalctl to display the complete log messages starting with the oldest entries. Refer to man 1 journalctl for options such as applying filters or changing the output format.

You can save the current state of systemd to a named snapshot and later revert to it with the isolate subcommand. This is useful when testing services or custom targets, because it allows you to return to a defined state at any time. A snapshot is only available in the current session and will automatically be deleted on reboot. A snapshot name must end in .snapshot.

With systemd, kernel modules can automatically be loaded at boot time via a configuration file in /etc/modules-load.d. The file should be named MODULE.conf and have the following content:

# load module MODULE at boot time
MODULE

In case a package installs a configuration file for loading a kernel module, the file gets installed to /usr/lib/modules-load.d. If two configuration files with the same name exist, the one in /etc/modules-load.d tales precedence.

For more information, see the modules-load.d(5) man page.

With System V init actions that need to be performed before loading a service, needed to be specified in /etc/init.d/before.local . This procedure is no longer supported with systemd. If you need to do actions before starting services, do the following:

On a traditional System V init system, it is not always possible to match a process to the service that spawned it. Some services, such as Apache, spawn a lot of third-party processes (for example, CGI or Java processes), which themselves spawn more processes. This makes a clear assignment difficult or even impossible. Additionally, a service may not finish correctly, leaving certain children alive.

systemd solves this problem by placing each service into its own cgroup. cgroups are a kernel feature that allows aggregating processes and all their children into hierarchical organized groups. systemd names each cgroup after its service. Since a non-privileged process is not allowed to “leave” its cgroup, this provides an effective way to label all processes spawned by a service with the name of the service.

To list all processes belonging to a service, use the command systemd-cgls, for example:

root # systemd-cgls --no-pager
├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 20
├─user.slice
│ └─user-1000.slice
│   ├─session-102.scope
│   │ ├─12426 gdm-session-worker [pam/gdm-password]
│   │ ├─15831 gdm-session-worker [pam/gdm-password]
│   │ ├─15839 gdm-session-worker [pam/gdm-password]
│   │ ├─15858 /usr/lib/gnome-terminal-server

[...]

└─system.slice
  ├─systemd-hostnamed.service
  │ └─17616 /usr/lib/systemd/systemd-hostnamed
  ├─cron.service
  │ └─1689 /usr/sbin/cron -n
  ├─ntpd.service
  │ └─1328 /usr/sbin/ntpd -p /var/run/ntp/ntpd.pid -g -u ntp:ntp -c /etc/ntp.conf
  ├─postfix.service
  │ ├─ 1676 /usr/lib/postfix/master -w
  │ ├─ 1679 qmgr -l -t fifo -u
  │ └─15590 pickup -l -t fifo -u
  ├─sshd.service
  │ └─1436 /usr/sbin/sshd -D

[...]

See Chapter 9, Kernel Control Groups for more information about cgroups.

As explained in Section 14.6.6, “Kernel control groups (cgroups)”, it is not always possible to assign a process to its parent service process in a System V init system. This makes it difficult to stop a service and its children. Child processes that have not been terminated remain as zombie processes.

systemd's concept of confining each service into a cgroup makes it possible to identify all child processes of a service and therefore allows you to send a signal to each of these processes. Use systemctl kill to send signals to services. For a list of available signals refer to man 7 signals.

The D-Bus service is the message bus for communication between systemd clients and the systemd manager that is running as pid 1. Even though dbus is a stand-alone daemon, it is an integral part of the init infrastructure.

Stopping dbus or restarting it in the running system is similar to an attempt to stop or restart PID 1. It breaks the systemd client/server communication and makes most systemd functions unusable.

Therefore, terminating or restarting dbus is neither recommended nor supported.

Updating the dbus or dbus-related packages requires a reboot. When in doubt whether a reboot is necessary, run the sudo zypper ps -s. If dbus appears among the listed services, you need to reboot the system.

Keep in mind that dbus is updated even when automatic updates are configured to skip the packages that require reboot.

By default, systemd is not overly verbose. If a service was started successfully, no output is produced. In case of a failure, a short error message is displayed. However, systemctl status provides a means to debug the start-up and operation of a service.

systemd comes with its own logging mechanism (“The Journal”) that logs system messages. This allows you to display the service messages together with status messages. The status command works similar to tail and can also display the log messages in different formats, making it a powerful debugging tool.

Timer units can use monotonic and realtime timers.

Every timer unit must have a corresponding systemd unit file it controls. In other words, a .timer file activates and manages the corresponding .service file. When used with a timer, the .service file does not require an [Install] section, as the service is managed by the timer.

To understand the basics of systemd timer units, we set up a timer that triggers the foo.sh shell script.

First step is to create a systemd service unit that controls the shell script. To do this, open a new text file for editing and add the following service unit definition:

[Unit]
Description="Foo shell script"

[Service]
ExecStart=/usr/local/bin/foo.sh

Save the file under the name foo.service in the directory /etc/systemd/system/.

Next, open a new text file for editing and add the following timer definition:

[Unit]
Description="Run foo shell script"

[Timer]
OnBootSec=5min
OnUnitActiveSec=24h
Unit=foo.service

[Install]
WantedBy=multi-user.target

The [Timer] section in the example above specifies what service to trigger (foo.service) and when to trigger it. In this case, the option OnBootSec specifies a monotonic timer that triggers the service five minutes after the system boot, while the option OnUnitActiveSec triggers the service 24 hours after the service has been activated (that is, the timer triggers the service once a day). Finally, the option WantedBy specifies that the timer should start when the system has reached the multi-user target.

Instead of a monotonic timer, you can specify a real-time one using the option OnCalendar. The following realtime timer definition triggers the related service unit once a week, starting on Monday at 12:00.

[Timer]
OnCalendar=weekly
Persistent=true

The option Persistent=true indicates that the service is triggered immediately after the timer activation if the timer missed the last start time (for example, because of the system being powered off).

The option OnCalendar can also be used to define specific dates times for triggering a service using the following format: DayOfWeek Year-Month-Day Hour:Minute:Second. The example below triggers a service at 5am every day:

OnCalendar=*-*-* 5:00:00

You can use an asterisk to specify any value, and commas to list possible values. Use two values separated by .. to indicate a contiguous range. The following example triggers a service at 6pm on Friday of every month:

OnCalendar=Fri *-*-1..7 18:00:00

To trigger a service at different times, you can specify several OnCalendar entries:

OnCalendar=Mon..Fri 10:00
OnCalendar=Sat,Sun 22:00

In the example above, a service is triggered at 10am on week days and at 10pm on weekends.

When you are done editing the timer unit file, save it under the name foo.timer in the /etc/systemd/system/ directory. To check the correctness of the created unit files, run the following command:

tux > sudo  systemd-analyze verify /etc/systemd/system/foo.*

If the command returns no output, the files have passed the verification successfully.

To start the timer, use the command sudo systemctl start foo.timer. To enable the timer on boot, run the command sudo systemctl enable foo.timer.

Since timers are treated as regular systemd units, you can manage them using systemctl. You can start a timer with systemctl start, enable a timer with systemctl enable, and so on. Additionally, you can list all active timers using the command systemctl list-timers. To list all timers, including inactive ones, run the command systemctl list-timers --all.