Setup Guide #

If your business can respond more quickly to new information and changing market conditions, you have a distinct advantage over those that cannot. Running your time-sensitive mission-critical applications using SUSE Linux Enterprise Real Time reduces process dispatch latencies and gives you the time advantage you need to increase profits, or avoid further financial losses, ahead of your competitors.

Keep the following points in mind:

To install SUSE Linux Enterprise Real Time 15 SP6, proceed as follows:

The following sections provide a brief introduction to the tools and possibilities of SUSE Linux Enterprise Real Time.

In some circumstances, it is beneficial to be able to run specific tasks only on defined CPUs. For this reason, the Linux kernel provides a feature called cpuset. The cpuset feature provides the means to do so-called “soft partitioning” of the system. This enables you to dedicate CPUs, together with some predefined memory, to work on particular tasks.

Note: CPUs, processors, and cores

Modern servers are typically built around multi-core CPUs, which means that a single processor socket typically contains many separate processor units. For example, a low-end processor might have four cores, while a high-end one may have from tens to hundreds of cores.

Secondarily, some vendors support simultaneous multithreading (SMT), which enables a single core to support two or more execution threads which can be partially overlapped. The processor makes this visible to the operating system by reporting each threads as an additional core.

The cpuset feature works on the level of logical processors: individual cores or SMT units, not processor sockets. When this document refers to “a CPU”, this denotes a logical processor.

cset consists of one “super command” called shield and the “regular commands” set and proc. The purpose of the super command shield is to create a common CPU shielding setup within one step by combining regular commands.

For more information about the options and parameters of the shield subcommand, view its help by running:

cset help shield

cset shield --cpu=3

cset shield --cpu=1,3,5-7

cset shield --cpu=1,3-7

cset shield --cpu=3-7

cset shield
cset: --> shielding system active with
cset: "system" cpuset of: 0-2 cpu, with: 47
cset: "user" cpuset of:  3-7 cpu,  with: 0

3.4 Shielding processes #

  

After a shielded CPU set is created, certain processes or groups of processes can be assigned to the shielded cpuset. To start a new process in the shielded CPU set, use the --exec option:

cset shield --exec APPLICATION

To move already-running processes to the shielded CPU set, use the --shield and --pid options. The --pid option accepts a comma-separated list of PIDs and range specifications:

cset shield --shield --pid=1,2,600-700

This moves processes with PID 1, 2, and from 600 to 700 to the shielded CPU set. If there is a gap in the range from 600 to 700, then only those available process will be moved to the shield without warning. The cset command handles threads like processes and will also interpret TIDs and assign them to the required CPU set.

Warning

The --shield option does not check the processes you request to move into the shield. This means that the command will move any processes that are bound to specific CPUs—even kernel threads. You can cause a complete system lockup by indiscriminately specifying arbitrary PIDs to the --shield command.

cset shield --verbose
cset: --> shielding system active with
cset: "system" cpuset of: 0-2,4-15 cpu, with:
   USER       PID  PPID S TASK NAME
      -------- ----- ----- - ---------
         root         1     0 S init [3]
[...]

cset: "user" cpuset of:    3 cpu, with: 1
   USER       PID  PPID S TASK NAME
      -------- ----- ----- - ---------
         root     10202 10170 S application

cset shield --unshield --pid=2,650-655

More detailed configuration of cpusets can be done with the cset commands set and proc.

The subcommand set is used to create, modify and destroy cpusets. Compared to the supercommand shield, the set subcommand can additionally assign memory nodes for NUMA machines.

Besides assigning memory nodes, the subcommand set creates cpusets in a tree-like structure, rooted at the root cpuset.

To create a cpuset with the subcommand set you need to specify the CPUs which should be used. Either use a comma-separated list or a range specification:

cset set --cpu=1-7 "/one"

This command will create a cpuset called one with assigned CPUs from #1 to #7. To specify a new cpuset called two that is a subset of one, proceed as follows:

cset set --cpu=6 "/one/two"

Cpusets follow certain rules. Children can only include CPUs that the parents already have. If you try to specify a different cpuset, the kernel cpuset subsystem will not let you create that cpuset. For example, if you create a cpuset that contains CPU3, and then attempt to create a child of that cpuset with a CPU other than 3, you will get an error, and the cpuset will not be created. The resulting error is somewhat cryptic and is usually “Permission denied”.

To show a table containing useful information, such as CPU lists and memory lists, use the -r parameter. The “-X” column shows the exclusive state of CPU or memory. The “path” column shows the real path in the virtual cpuset file system.

cset set -r

On NUMA machines, memory nodes can be assigned to a cpuset similar to CPUs. The --mem option of the subcommand set allows a comma-separated and inclusive range specification of memory nodes. This example will assign MEM1, MEM3, MEM4, MEM5 and MEM6 to the cpuset new_set:

cset set --mem=1,3-6 new_set

Additionally, with the --cpu_exclusive and --mem_exclusive options (without any additional arguments) set the CPUs or memory nodes exclusive to a cpuset:

cset set --cpu_exclusive "/one"

The status of exclusive state of CPU or memory is shown in the -X column when running:

cset set -r

For more detailed information about options and parameters of the subcommand set, view the cset help:

cset help set

After the cpuset is initialized, the subcommand proc can start processes on certain cpusets with the --exec option. The following will start the application fastapp within the cpuset new_set:

cset proc --exec --set new_set fastapp

To move an already-running process inside an already-existing cpuset, use the option --move. It accepts a comma-separated list and range specifications of PIDs. The following command will move processes with PID 2442 and within the range between 3000 and 3200 into the cpuset new_set:

cset proc --move 2442,3000-3200 new_set

Listing processes running within a specific cpuset can be done by using the option --list.

cset proc --list new_set

The subcommand proc can also move the entire list of processes within one cpuset to another cpuset by using the option --fromset and --toset. This will move all process assigned to old_set and assign them to new_set:

cset proc --move --fromset old_set \
   --toset new_set

For more detailed information about options and parameters of the subcommand proc, view the help:

cset help proc

Use the chrt command to manipulate the real-time attributes of an already-running process (such as scheduling policy and priority), or to execute a new process with specified real-time attributes.

It is highly recommended for applications which do not use real-time specific attributes by themselves, but should nevertheless experience the full advantages of real-time. To get full real-time experiences, call these applications with the chrt command and the right set of scheduler policy and priority parameters.

With the following command, all running processes with their real-time specific attributes are shown. The selection class shows the current scheduler policy and rtprio the real-time priority:

ps -eo pid,tid,class,rtprio,comm
...
 1437  1437 FF      40  fastapp

The truncated example above shows the fastapp process with PID 1437 running and with scheduler policy SCHED_FIFO and priority 40. Scheduler policy abbreviations are:

It is also possible to obtain the current scheduler policy, and the priority of single processes, by specifying the PID of the process with the -p parameter. For example:

chrt -p 1437

Scheduler policies have different minimum and maximum priority values. Minimum and maximum values for each available scheduler policy can be retrieved with chrt:

chrt -m

To change the scheduler policy and the priority of a running process, chrt provides the options --fifo for SCHED_FIFO, --rr for SCHED_RR and --other for SCHED_OTHER. The following example will change the scheduler policy to SCHED_FIFO with priority 42 for PID 1437:

chrt --fifo -p 42 1437

Warning

Handle the changing of real-time attributes of processes with care. Increasing the priority of certain processes can harm the entire system, depending on the behavior of the process. In some cases, this can lead to a complete system lockup or bad influences on certain devices.

For more information about chrt, see the chrt man page with man 1 chrt.

The default behavior of the kernel is to keep a process running on the same CPU if the system load is balanced over the available CPUs. Otherwise, the kernel tries to improve the load balancing by moving processes to an idling CPU. In some situations, however, it is desirable to set a CPU affinity for a given process. In this case, the kernel will not move the process away from the selected CPUs. For example, if you use shielding, the shielded CPUs will not run any processes that do not have an affinity to the shielded CPUs. Another possibility to remove load from the other CPUs is to run all low priority tasks on a selected CPU.

If a task is running inside a specific cpuset, the affinity dialog must match at least one of the CPUs available in this set. The taskset command will not move a process outside the cpuset it is running in.

To set or retrieve the CPU affinity of a task, use a bitmask. This mask is represented by a hexadecimal number. If you count the bits of this bitmask, the lowest bit represents the first logical CPU as found in /proc/cpuinfo. For example:

If a given dialog does not contain any valid CPU on the system, the taskset command will return an error. If taskset returns without an error, the given program has been scheduled to the specified list of CPUs.

The command taskset starts a new process with a given CPU affinity, or to redefine the CPU affinity of an already running process.

For more detailed information about taskset, see the man page man 1 taskset.

Handling I/O is one of the critical issues for all high-performance systems. If a task has lots of CPU power available, but must wait for the disk, it will not work as efficiently as it could. The Linux kernel provides three different scheduling classes to determine the I/O handling for a process. All of these classes can be fine-tuned with a nice level.

To change I/O schedulers and nice values, use the ionice command. This provides a means to tune the scheduler of already-running processes, or to start new processes with specific I/O settings.

For more detailed information about ionice, see the ionice man page with man 1 ionice

The Linux kernel provides several block device schedulers that can be selected individually for each block device. All but the noop scheduler perform a kind of ordering of requested blocks to reduce head movements on the hard disk. If you use an external storage system that has its own scheduler, you should disable the Linux internal reordering by selecting the noop scheduler.

To print the current scheduler of a block device such as /dev/sda, use the following command:

cat /sys/block/sda/queue/scheduler
noop deadline [cfq]

In this case, the scheduler for /dev/sda is set to cfq, the Completely Fair Queuing scheduler. This is the default scheduler on SUSE Linux Enterprise Real Time.

To change the schedulers, echo one of the names noop, deadline, or cfq into /sys/block/<device>/scheduler. For example, if you want to set the I/O scheduler of the device /dev/sda to noop, use the following command:

echo "noop" > /sys/block/sda/queue/scheduler

To set other variables in the /sys file system, use a similar approach.

All schedulers, except for the noop scheduler, have several common parameters that may be tuned for each block device. You can access these parameters with sysfs in the /sys/block/<device>/queue/iosched/ directory. The following parameters are tuneable for the respective scheduler:

The system default Block Device I/O Scheduler can be also set by the kernel parameter elevator=. For example, elevator=deadline changes the I/O Scheduler to deadline.

A lot of information about real-time implementations and administration can be found on the Internet. The following list contains several selected links: