10 Kernel control groups #

It became apparent, in practice, that there is not a single default that applies to all use cases. SUSE Linux Enterprise Server ships with two custom configurations that override the upstream defaults for system units and for user slices, and sets them both to infinity. /usr/lib/systemd/system.conf.d/__25-defaults-SLE.conf contains these lines:

[Manager]
DefaultTasksMax=infinity

/usr/lib/systemd/system/user-.slice.d/25-defaults-SLE.conf contains these lines:

[Slice]
TasksMax=infinity

Use systemctl to verify the DefaultTasksMax value:

> systemctl show --property DefaultTasksMax
DefaultTasksMax=infinity

infinity means having no limit. It is not a requirement to change the default, but setting some limits may help to prevent system crashes from runaway processes.

Change the global DefaultTasksMax value by creating a new override file, /etc/systemd/system.conf.d/90-system-tasksmax.conf, and write the following lines to set a new default limit of 256 tasks per system unit:

[Manager]
DefaultTasksMax=256

Load the new setting, then verify that it changed:

> sudo systemctl daemon-reload
> systemctl show --property DefaultTasksMax
DefaultTasksMax=256

Adjust this default value to suit your needs. You can set different limits on individual services as needed. This example is for MariaDB. First check the current active value:

> systemctl status mariadb.service
  ● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset>
   Active: active (running) since Tue 2020-05-26 14:15:03 PDT; 27min ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
 Main PID: 11845 (mysqld)
   Status: "Taking your SQL requests now..."
    Tasks: 30 (limit: 256)
   CGroup: /system.slice/mariadb.service
           └─11845 /usr/sbin/mysqld --defaults-file=/etc/my.cnf --user=mysql

The Tasks line shows that MariaDB currently has 30 tasks running, and has an upper limit of the default 256, which is inadequate for a database. The following example demonstrates how to raise MariaDB's limit to 8192.

> sudo systemctl set-property mariadb.service TasksMax=8192
> systemctl status mariadb.service
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disab>
  Drop-In: /etc/systemd/system/mariadb.service.d
           └─50-TasksMax.conf
   Active: active (running) since Tue 2020-06-02 17:57:48 PDT; 7min ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 3446 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper upgrade (code=exited, sta>
  Process: 3440 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper install (code=exited, sta>
 Main PID: 3452 (mysqld)
   Status: "Taking your SQL requests now..."
    Tasks: 30 (limit: 8192)
   CGroup: /system.slice/mariadb.service
           └─3452 /usr/sbin/mysqld --defaults-file=/etc/my.cnf --user=mysql

systemctl set-property applies the new limit and creates a drop-in file for persistence, /etc/systemd/system/mariadb.service.d/50-TasksMax.conf, that contains only the changes you want to apply to the existing unit file. The value does not have to be 8192, but should be whatever limit is appropriate for your workloads.

The default limit on users should be fairly high, because user sessions need more resources. Set your own default for any user by creating a new file, for example /etc/systemd/system/user-.slice.d/40-user-taskmask.conf. The following example sets a default of 16284:

[Slice]
TasksMax=16284

Note: Numeric prefixes reference

See Section 19.5.3, “Creating drop-in files manually” to learn what numeric prefixes are expected for drop-in files.

Then reload systemd to load the new value, and verify the change:

> sudo systemctl daemon-reload
> systemctl show --property TasksMax user-1000.slice
TasksMax=16284

How do you know what values to use? This varies according to your workloads, system resources, and other resource configurations. When your TasksMax value is too low, you will see error messages such as Failed to fork (Resources temporarily unavailable), Can't create thread to handle new connection, and Error: Function call 'fork' failed with error code 11, 'Resource temporarily unavailable'.

For more information on configuring system resources in systemd, see systemd.resource-control (5).

The following subsections describe steps that you must take in advance when you design and configure your system, since those aspects cannot be changed during runtime.

> cat /sys/class/block/sda/queue/scheduler
mq-deadline kyber bfq [none]

 # echo bfq > /sys/class/block/sda/queue/scheduler

r
`-  a      IOWeight=100
    `- [c] IOWeight=300
    `-  d  IOWeight=100
`- [b]     IOWeight=200

You can apply the values to (long running) services permanently.

> sudo systemctl set-property fast.service IOWeight=400
> sudo systemctl set-property slow.service IOWeight=50
> sudo systemctl set-property throttled.service IOReadBandwidthMax="/dev/sda 1M"

Alternatively, you can apply I/O control to individual commands, for example:

> sudo systemd-run --scope -p IOWeight=400 high_prioritized_command
> sudo systemd-run --scope -p IOWeight=50 low_prioritized_command
> sudo systemd-run --scope -p IOReadBandwidthMax="/dev/sda 1M" dd if=/dev/sda of=/dev/null bs=1M count=10

The following list items describe I/O control behavior, and what you should expect under various conditions.