2 The basic shielding model #
Although any setup of cpusets can really be described as
shielding, there is one prevalent shielding model in
use that is so common that cset has a subcommand that is dedicated to its
use. This subcommand is called shield.
The concept behind this model is the use of three cpusets:
Root
cpuset. is always present in all configurations and contains all CPUs.System
cpuset. contains CPUs which are used for system tasks. These are the normal tasks that are not important, but which need to run on the system.User
cpuset. “the shield”, contains CPUs which are used for important tasks. Only those tasks that are somehow important, usually tasks whose performance determines the overall rating for the machine, are run in theusercpuset.
The shield subcommand manages all of these cpusets and
lets you define the CPUs and memory nodes that are in the
shielded and unshielded sets.
The subcommand automatically moves all movable tasks on the system into the
unshielded cpuset on shield activation, and back into
the root cpuset on shield tear down. The subcommand
lets you move tasks into and out of the shield. Kernel threads are excluded
from these migrations.
The shield subcommand abstracts the management of these cpusets away from
you. It provides options that drive how the shield is set up, which tasks
are to be shielded or not, and the status of the shield. In
fact, you need not be bothered with the naming of the required cpusets or
even where the cpuset file system is mounted. cset and
the shield subcommand takes care of all that.
If you need to define more cpusets for your application, it is likely
that this simple shielding is not rich enough for you. In this
case, you should transition to using the set and
proc subcommands described in
Chapter 4, Full-featured cpuset manipulation commands.
2.1 A simple shielding example #
Assume a four-core machine that has uniform memory access. This means there are four CPUs at your disposal and there is only one memory node available. On such machines, there is no need to specify any memory node parameters to cset, it sets up the only available memory node by default.
Usually, one wants to dedicate as many CPUs to the shield as possible and leave a minimal set of CPUs for normal system processing. The reasoning for this is, the performance of the important tasks will rule the performance of the installation as a whole. These important tasks need as many resources available to them as possible, exclusive of other, unimportant tasks that are running on the system.
In this document task is used to represent either a process or a thread that is running on the system.
2.2 Setup and teardown of the shield #
To set up a shield of three CPUs with one CPU left for low priority system processing, issue the following command.
tux >csetshield -c 1-3 cset: --> activating shielding: cset: moving 176 tasks from root into system cpuset... [==================================================]% cset: "system" cpuset of CPUSPEC(0) with 176 tasks running cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running
This command does several things. First, it creates a
user cpuset with what is called a CPUSPEC (CPU
specification) from the -c/--cpu option. This CPUSPEC
specifies to use CPUs 1 through 3 inclusively. Next, the command creates
a system cpuset with a CPUSPEC that is the inverse of
the -c option for the current machine. On this machine
that cpuset will only contain the first CPU, CPU0. Next, all user space
processes running in the root cpuset are transferred
to the system cpuset. This makes all those processes
run only on CPU0. The effect of this is that the shield consists of CPUs
1 through 3 and they are now idling.
Note that the command did not move the kernel threads that are running
in the root cpuset to the system
cpuset. This is because you may want these kernel threads to use all
available CPUs.
The shield setup command above outputs the information of which cpusets
were created and how many tasks are running on each. To see
the current status of the shield again, issue this command:
tux >csetshield cset: --> shielding system active with cset: "system" cpuset of CPUSPEC(0) with 176 tasks running cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running
Which shows us that the shield is set up and that 176 tasks are running
in the system cpuset—the
unshielded cpuset.
It is important to move all possible tasks from the
root cpuset to the unshielded
system cpuset because a task’s cpuset property is
inherited by its children. As all running tasks (including init) have been
moved to the unshielded system cpuset,
that means that any new tasks that are spawned will also run in the
unshielded system cpuset.
Note. There is a minor chance that a task forks during move and its
child remains in the root cpuset.
Kernel threads can be both unbound or bound to specific CPUs.
If a kernel thread is bound to a specific
CPU, then it is generally not a good idea to move that thread to the
system set because at worst it may hang the system
and at best it will slow the system down significantly. These threads
are usually the IRQ threads on a real time Linux kernel, for example,
and you should not move these kernel threads into
system. If you leave them in the
root cpuset, then they will have access to all CPUs.
However, if your application demands an even “quieter”
shield, you should look at isolcpus= kernel command line
argument.
You can get a detailed listing of what is running in the shield by
adding either -s/--shield or
-u/--unshield to the shield
subcommand and using the verbose flag. You will get output similar to
the following.
tux >csetshield --unshield -v cset: "system" cpuset of CPUSPEC(0) with 251 tasks running USER PID PPID SPPr TASK NAME -------- ----- ----- ---- --------- root 1 0 Soth init [5] root 2 0 Soth [kthreadd] root 84 2 Sf50 [IRQ-9 ]... tux 31796 31789 Soth less root 32653 25222 Roth python ./cset shield --unshield -v
The previous listing is abbreviated—there are 251 tasks running in
the system set. However, the SPPr
field may need a little
explanation. SPPr stands for State, Policy and
Priority. You can see that the initial two tasks are Stopped
and running in timeshare priority, marked as oth (for
other). The [IRQ-9] task is also stopped, but marked
at real time FIFO policy with a priority of 50. The last task in the
listing is the cset command itself and is marked as
running. Also note that adding a second -v/--verbose
option will not restrict the output to fit into an 80 character screen.
Tear down of the shield, stopping the shield in other words, is done
with the -r/--reset option to the shield subcommand.
When this command is issued, both the system and
user cpusets are deleted and any tasks that are
running in both of those cpusets are moved to the
root cpuset. Once so moved, all tasks will have
access to all resources on the system. For example:
tux >csetshield --reset cset: --> deactivating/reseting shielding cset: moving 0 tasks from "/user" user set to root set... cset: moving 250 tasks from "/system" system set to root set... [==================================================]% cset: deleting "/user" and "/system" sets cset: done
2.3 Moving interesting tasks into and out of the shield #
Now that a shield is running, the objective is to run processes that you have categorized as important in that shield. These processes can be anything, but usually they are directly related to the purpose of the machine. There are two ways to run tasks in the shield:
Execute a process into the shield
Move an already running task into the shield
2.3.1 Executing a process into the shield #
Running a new process in the shield can be done with the
-e/--exec option to the shield
subcommand. This is the simplest way to get a task to run in the
shield. For this example, execute a new Bash shell into the shield
with the following commands.
tux >csetshield -s cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running cset: donetux >csetshield -e bash cset: --> last message, executed args into cpuset "/user", new pid is: 13300tux >csetshield -s -v cset: "user" cpuset of CPUSPEC(1-3) with 2 tasks running USER PID PPID SPPr TASK NAME -------- ----- ----- ---- --------- root 13300 8583 Soth bash root 13329 13300 Roth python ./cset shield -s -vtux >exittux >csetshield -s cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running cset: done
The first command above lists the status of the shield. You see that the shield is defined as CPUs 1 through 3 inclusive and currently there are no tasks running in it.
The second command executes the Bash shell into the shield with the
-e option. The last message of cset
lists the PID of the new process.
cset command
cset follows the tradition of separating the tool
options from the command to be executed options with a double hyphen
(--). This is not shown in this simple example, but
if the command you want to execute also takes options, separate them with
the double hyphen as follows:
tux >csetshield -e mycommand -- -v
The -v will be passed to mycommand, and not to cset.
The next command lists the status of the shield again. There are two
tasks running shielded: our new shell and the cset status command
itself. Remember that the cpuset property of a task is inherited by its
children. Since running the new shell in the shield, its child, which is
the status command, also ran in the shield.
Executing a shell into a shield is a useful way to experiment with running tasks in the shield since all children of the shell will also run in the shield.
The last command exits the shell. After this, shield status is requested again but again, it does not contain any tasks.
You may have noticed in the output above that both the new shell and
the status command are running as the root user. This is
because cset needs to run as root and so all it is children
will also run as root. If you need to run a process under a
different user and or group, you may use the --user
and --group options for execution as follows.
tux >csetshield --user=tux --group=users -e bash cset: --> last message, executed args into cpuset "/user", new pid is: 14212tux >csetshield -s -v cset: "user" cpuset of CPUSPEC(1-3) with 2 tasks running USER PID PPID SPPr TASK NAME -------- ----- ----- ---- --------- tux 14212 8583 Soth bash tux 14241 14212 Roth python ./cset shield -s -v
2.3.2 Moving a running task into and out of the shield #
While executing a process into the shield is undoubtedly useful, most of
the time, you will want to move already running tasks into and out of
the shield. The cset shield subcommand includes two
options for doing this: -s/--shield and
-u/--unshield. These options require a PIDSPEC
(process specification) to also be specified with the
-p/--pid option. The PIDSPEC defines which tasks get
operated on. The PIDSPEC can be a single process ID, a list of process
IDs separated by commas, and a list of process ID ranges separated by
dashes, groups of which are separated by commas. For example:
--shield --pid1234This PIDSPEC argument specifies that PID
1234be shielded.--shield --pid1234,42,1934,15000,15001,15002This PIDSPEC argument specifies that this list of PIDs only be moved into the shield.
--unshield -p5000,5100,6010-7000,9232This PIDSPEC argument specifies that PIDs
5000,5100and9232be unshielded (moved out of the shield) along with any existing PID that is in the range6010through7000inclusive.
A range in a PIDSPEC does not need to have tasks running for every number in that range. In fact, it is not even an error if there are no tasks running in that range: none will be moved in that case. The range only specifies to act on any tasks that have a PID or TID that is within that range.
Use of the appropriate PIDSPEC can thus be handy to move tasks and
groups of tasks into and out of the shield. Additionally, there is one
more option that can help with multi-threaded processes, and that is
the --threads flag. If this flag is used together with a
shield or unshield command with a
PIDSPEC and if any of the task IDs in the PIDSPEC belong to a thread in
a process container, then all the
sibling threads in that process container will get shielded or
unshielded as well. This flag provides an easy mechanism to
shield/unshield all threads of a process by simply specifying one
thread in that process.
The following example moves the current shell into the shield with a range PIDSPEC and back out with the Bash variable for the current PID.
tux >echo$$ 22018tux >csetshield -s -p 22010-22020 cset: --> shielding following pidspec: 22010-22020 cset: donetux >csetshield -s -v cset: "user" cpuset of CPUSPEC(1-3) with 2 tasks running USER PID PPID SPPr TASK NAME -------- ----- ----- ---- --------- root 3770 22018 Roth python ./cset shield -s -v root 22018 5034 Soth bash cset: donetux >csetshield -u -p $$ cset: --> unshielding following pidspec: 22018 cset: donetux >csetshield -s cset: "user" cpuset of CPUSPEC(1-3) with 0 tasks running cset: done