6 Slurm — utility for HPC workload management #

On the management server, install the slurm package with the command zypper in slurm.

On the compute nodes, install the slurm-node package with the command zypper in slurm-node.

On the management server and the compute nodes, the package munge is installed automatically. Configure, enable and start MUNGE on the management server and compute nodes as described in Section 4.4, “MUNGE authentication”. Ensure that the same munge key is shared across all nodes.

On the management server, edit the main configuration file /etc/slurm/slurm.conf:

1. Configure the parameter SlurmctldHost=SLURMCTLD_HOST with the host name of the management server.

   To find the correct host name, run hostname -s on the management server.

2. Under the COMPUTE NODES section, add the following lines to define the compute nodes:

   NodeName=NODE_LIST State=UNKNOWN
   PartitionName=normal Nodes=NODE_LIST Default=YES MaxTime=24:00:00 State=UP

   Replace NODE_LIST with the host names of the compute nodes, either comma-separated or as a range (for example: node[1-100]).

   The NodeName line also allows specifying additional parameters for the nodes, such as Boards, SocketsPerBoard CoresPerSocket, ThreadsPerCore, or CPU. The actual values of these can be obtained by running the following command on the compute nodes:

   node1# slurmd -C

Copy the modified configuration file /etc/slurm/slurm.conf from the management server to all compute nodes:

management# scp /etc/slurm/slurm.conf COMPUTE_NODE:/etc/slurm/

On the management server, start slurmctld and enable it so that it starts on every boot:

management# systemctl enable --now slurmctld.service

On each compute node, start slurmd and enable it so that it starts on every boot:

node1# systemctl enable --now slurmd.service

Check the status and availability of the compute nodes by running the sinfo command. You should see output similar to the following:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
normal*      up 1-00:00:00      2   idle node[01-02]

If the node state is not idle, see Section 6.4, “Frequently asked questions”.

Test the Slurm installation by running the following command:

management# srun sleep 30

This runs the sleep command on a free compute node for 30 seconds.

In another shell, run the squeue command during the 30 seconds that the compute node is asleep. You should see output similar to the following:

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
    1    normal    sleep     root  R       0:05      1 node02

Create the following shell script and save it as sleeper.sh:

#!/bin/bash
echo "started at $(date)"
sleep 30
echo "finished at $(date)"

Run the shell script in the queue:

management# sbatch sleeper.sh

The shell script is executed when enough resources are available, and the output is stored in the file slurm-${JOBNR}.out.

Install the MariaDB SQL database:

management# zypper install mariadb

Start and enable MariaDB:

management# systemctl enable --now mariadb

Secure the database:

management# mysql_secure_installation

Connect to the SQL database:

management# mysql -u root -p

Create the Slurm database user and grant it permissions for the Slurm database, which will be created later:

mysql> create user 'slurm'@'localhost' identified by 'PASSWORD';
mysql> grant all on slurm_acct_db.* TO 'slurm'@'localhost';

You can choose to use a different user name or database name. In this case, you must also change the corresponding values in the /etc/slurm/slurmdbd.conf file later.

Exit the database:

mysql> exit

Install the slurmdbd package:

management# zypper in slurm-slurmdbd

Edit the /etc/slurm/slurmdbd.conf file so that the daemon can access the database. Change the following line to the password that you used in Step 5:

StoragePass=password

If the database is on a different node, or if you chose a different user name or database name, you must also modify the following lines:

StorageUser=slurm
StorageLoc=slurm_acct_db
DbdAddr=localhost
DbdHost=localhost

Start and enable slurmdbd:

management# systemctl enable --now slurmdbd

The first start of slurmdbd might take some time.

To enable accounting, edit the /etc/slurm/slurm.conf file to add the connection between slurmctld and the slurmdbd daemon. Ensure that the following lines appear as shown:

JobAcctGatherType=jobacct_gather/linux
JobAcctGatherFrequency=30
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=localhost

This example assumes that slurmdbd is running on the same node as slurmctld. If not, change localhost to the host name or IP address of the node where slurmdbd is running.

Make sure slurmdbd is running before you continue:

management# systemctl status slurmdbd

If you restart slurmctld before slurmdbd is running, slurmctld fails because it cannot connect to the database.

Restart slurmctld:

management# systemctl restart slurmctld

This creates the Slurm database and adds the cluster to the database (using the ClusterName from /etc/slurm/slurm.conf).

(Optional) By default, Slurm does not take any group membership into account, and the system groups cannot be mapped to Slurm. However, you can mimic system groups with accounts. In Slurm, accounts are usually entities billed for cluster usage, while users identify individual cluster users. Multiple users can be associated with a single account.

The following example creates an umbrella group bavaria for two subgroups called nuremberg and munich:

management# sacctmgr add account bavaria \
Description="umbrella group for subgroups" Organization=bavaria

management# sacctmgr add account nuremberg,munich parent=bavaria \
Description="subgroup" Organization=bavaria

The following example adds a user called tux to the subgroup nuremberg:

management# sacctmgr add user tux Account=nuremberg

Stop the slurmdbd service:

DBnode# rcslurmdbd stop

Ensure that slurmdbd is not running anymore:

DBnode# rcslurmdbd status

slurmctld might remain running while the database daemon is down. During this time, requests intended for slurmdbd are queued internally. The DBD Agent Queue size is limited, however, and should therefore be monitored with sdiag.

Create a backup of the slurm_acct_db database:

DBnode# mysqldump -p slurm_acct_db > slurm_acct_db.sql

If needed, this can be restored with the following command:

DBnode# mysql -p slurm_acct_db < slurm_acct_db.sql

During the database conversion, the variable innodb_buffer_pool_size must be set to a value of 128 MB or more. Check the current size:

DBnode# echo  'SELECT @@innodb_buffer_pool_size/1024/1024;' | \
mysql --password --batch

If the value of innodb_buffer_pool_size is less than 128 MB, you can change it for the duration of the current session (on mariadb):

DBnode# echo 'set GLOBAL innodb_buffer_pool_size = 134217728;' | \
mysql --password --batch

Alternatively, to permanently change the size, edit the /etc/my.cnf file, set innodb_buffer_pool_size to 128 MB, then restart the database:

DBnode# rcmysql restart

If you need to update MariaDB, run the following command:

DBnode# zypper update mariadb

Convert the database tables to the new version of MariaDB:

DBnode# mysql_upgrade --user=root --password=ROOT_DB_PASSWORD;

Install the new version of slurmdbd:

DBnode# zypper install --force-resolution slurm_VERSION-slurmdbd

Rebuild the database. If you are using a backup slurmdbd, perform this step on the primary slurmdbd first.

Because a conversion might take a considerable amount of time, the systemd service might time out during the conversion. Therefore, we recommend performing the migration manually by running slurmdbd from the command line in the foreground:

DBnode# /usr/sbin/slurmdbd -D -v

When you see the following message, you can shut down slurmdbd by pressing Ctrl–C:

Conversion done:
success!

Before restarting the service, remove or replace any deprecated configuration options. Check the deprecated options in the Release Notes.

Restart slurmdbd:

DBnode# systemctl start slurmdbd

Note: No daemonization during rebuild

During the rebuild of the Slurm database, the database daemon does not daemonize.

Back up the configuration file/etc/slurm/slurm.conf. Because this file should be identical across the entire cluster, it is sufficient to do so only on the main management server.

On the main management server, edit /etc/slurm/slurm.conf and set SlurmdTimeout and SlurmctldTimeout to sufficiently high values to avoid timeouts while slurmctld and slurmd are down:

SlurmctldTimeout=3600
SlurmdTimeout=3600

We recommend at least 60 minutes (3600), and more for larger clusters.

Copy the updated /etc/slurm/slurm.conf from the management server to all nodes:

1. Obtain the list of partitions in /etc/slurm/slurm.conf.

2. Copy the updated configuration to the compute nodes:

   management# cp /etc/slurm/slurm.conf /etc/slurm/slurm.conf.update
   management# sudo -u slurm /bin/bash -c 'cat /etc/slurm/slurm.conf.update | \
   pdsh -R mrsh -P PARTITIONS "cat > /etc/slurm/slurm.conf"'
   management# rm /etc/slurm/slurm.conf.update

3. Reload the configuration file on all compute nodes:

   management# scontrol reconfigure

4. Verify that the reconfiguration took effect:

   management# scontrol show config | grep Timeout

Shut down all running slurmctld instances, first on any backup management servers, and then on the main management server:

management# systemctl stop slurmctld

Back up the slurmctld state files. slurmctld maintains persistent state information. Almost every major version involves changes to the slurmctld state files. This state information is upgraded if the upgrade remains within the supported version range and no data is lost.

However, if a downgrade is necessary, state information from newer versions is not recognized by an older version of slurmctld and is discarded, resulting in a loss of all running and pending jobs. Therefore, back up the old state in case an update needs to be rolled back.

1. Determine the StateSaveLocation directory:

   management# scontrol show config | grep StateSaveLocation

2. Create a backup of the content of this directory. If a downgrade is required, restore the content of the StateSaveLocation directory from this backup.

Shut down slurmd on the compute nodes:

management# pdsh -R ssh -P PARTITIONS systemctl stop slurmd

Upgrade slurmctld on the main and backup management servers:

management# zypper install --force-resolution slurm_VERSION

Important: Upgrade all Slurm packages at the same time

If any additional Slurm packages are installed, you must upgrade those as well. This includes:

• slurm-pam_slurm

• slurm-sview

• perl-slurm

• slurm-lua

• slurm-torque

• slurm-config-man

• slurm-doc

• slurm-webdoc

• slurm-auth-none

• pdsh-slurm

All Slurm packages must be upgraded at the same time to avoid conflicts between packages of different versions. This can be done by adding them to the zypper install command line described above.

Upgrade slurmd on the compute nodes:

management# pdsh -R ssh -P PARTITIONS \
zypper install --force-resolution slurm_VERSION-node

Note: Memory size seen by slurmd might change on update

Under certain circumstances, the amount of memory seen by slurmd might change after an update. If this happens, slurmctld puts the nodes in a drained state. To check whether the amount of memory seem by slurmd changed after the update, run the following command on a single compute node:

node1# slurmd -C

Compare the output with the settings in slurm.conf. If required, correct the setting.

Before restarting the service, remove or replace any deprecated configuration options. Check the deprecated options in the Release Notes.

If you replace deprecated options in the configuration files, these configuration files can be distributed to all management servers and compute nodes in the cluster by using the method described in Step 3.

Restart slurmd on all compute nodes:

management# pdsh -R ssh -P PARTITIONS systemctl start slurmd

Restart slurmctld on the main and backup management servers:

management# systemctl start slurmctld

Check the status of the management servers. On the main and backup management servers, run the following command:

management# systemctl status slurmctld

Verify that the services are running without errors. Run the following command to check whether there are any down, drained, failing, or failed nodes:

management# sinfo -R

Restore the original values of SlurmdTimeout and SlurmctldTimeout in /etc/slurm/slurm.conf, then copy the restored configuration to all nodes by using the method described in Step 3.