1. On the server where MUNGE is installed, check the permissions, owner, and file type of the key file /etc/munge/munge.key:

   > sudo stat --format "%F %a %G %U %n" /etc/munge/munge.key

   The settings should be as follows:

   400 regular file munge munge /etc/munge/munge.key

2. Calculate the MD5 sum of munge.key:

   > sudo md5sum /etc/munge/munge.key

3. Copy the key to the listed nodes using pdcp:

   > pdcp -R ssh -w NODELIST /etc/munge/munge.key /etc/munge/munge.key

4. Check the key settings on the remote nodes:

   > pdsh -R ssh -w HOSTLIST stat --format \"%F %a %G %U %n\" /etc/munge/munge.key
   > pdsh -R ssh -w HOSTLIST md5sum /etc/munge/munge.key

   Ensure that they match the settings on the MUNGE server.

1. On the management server, install the slurm package with the command zypper in slurm.

2. On the compute nodes, install the slurm-node package with the command zypper in slurm-node.

3. On the management server and the compute nodes, the package munge is installed automatically. Configure, enable and start MUNGE on the management server and compute nodes as described in Section 3.4, “MUNGE authentication”. Ensure that the same munge key is shared across all nodes.

1. On the management server, edit the main configuration file /etc/slurm/slurm.conf:

   1. Configure the parameter SlurmctldHost=SLURMCTLD_HOST with the host name of the management server.

      To find the correct host name, run hostname -s on the management server.

   2. Under the COMPUTE NODES section, add the following lines to define the compute nodes:

      NodeName=NODE_LIST State=UNKNOWN
      PartitionName=normal Nodes=NODE_LIST Default=YES MaxTime=24:00:00 State=UP

      Replace NODE_LIST with the host names of the compute nodes, either comma-separated or as a range (for example: node[1-100]).

      The NodeName line also allows specifying additional parameters for the nodes, such as Boards, SocketsPerBoard CoresPerSocket, ThreadsPerCore, or CPU. The actual values of these can be obtained by running the following command on the compute nodes:

      node1 # slurmd -C

2. Copy the modified configuration file /etc/slurm/slurm.conf from the management server to all compute nodes:

   management # scp /etc/slurm/slurm.conf COMPUTE_NODE:/etc/slurm/

3. On the management server, start slurmctld and enable it so that it starts on every boot:

   management # systemctl enable --now slurmctld.service

4. On each compute node, start slurmd and enable it so that it starts on every boot:

   node1 # systemctl enable --now slurmd.service

1. Check the status and availability of the compute nodes by running the sinfo command. You should see output similar to the following:

   PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
   normal*      up 1-00:00:00      2   idle node[01-02]

   If the node state is not idle, see Section 5.4, “Frequently asked questions”.

2. Test the Slurm installation by running the following command:

   management # srun sleep 30

   This runs the sleep command on a free compute node for 30 seconds.

   In another shell, run the squeue command during the 30 seconds that the compute node is asleep. You should see output similar to the following:

   JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
       1    normal    sleep     root  R       0:05      1 node02

3. Create the following shell script and save it as sleeper.sh:

   #!/bin/bash
   echo "started at $(date)"
   sleep 30
   echo "finished at $(date)"

   Run the shell script in the queue:

   management # sbatch sleeper.sh

   The shell script is executed when enough resources are available, and the output is stored in the file slurm-${JOBNR}.out.

Note: MariaDB

If you want to use an external SQL database (or you already have a database installed on the management server), you can skip Step 1 and Step 2.

1. Install the MariaDB SQL database with zypper in mariadb.

2. Start and enable MariaDB:

   management # systemctl enable --now mariadb

3. Secure the database with the command mysql_secure_installation.

4. Connect to the SQL database, for example with the command mysql -u root -p.

5. Create the Slurm user and the database with the following commands:

   sql > create user 'slurm'@'localhost' identified by 'PASSWORD';
   sql > grant all on slurm_acct_db.* TO 'slurm'@'localhost';
   sql > create database slurm_acct_db;

   After these steps are complete, exit the database.

6. Install the slurmdbd package:

   management # zypper in slurm-slurmdbd

7. Edit the /etc/slurm/slurmdbd.conf file so that the daemon can access the database. Change the following line to the password that you used in Step 5:

   StoragePass=password

   If you chose another location or user for the SQL database, you must also modify the following lines:

   StorageUser=slurm
   DbdAddr=localhost
   DbdHost=localhost

8. Start and enable slurmdbd:

   management # systemctl enable --now slurmdbd

   The first start of slurmdbd will take some time.

9. To enable accounting, edit the /etc/slurm/slurm.conf file to add the connection between slurmctld and the slurmdbd daemon. Ensure that the following lines appear as shown:

   JobAcctGatherType=jobacct_gather/linux
   JobAcctGatherFrequency=30
   AccountingStorageType=accounting_storage/slurmdbd
   AccountingStorageHost=localhost

   

   This example assumes that slurmdbd is running on the same node as slurmctld. If not, change localhost to the host name or IP address of the node where slurmdbd is running.

10. Ensure that a table for the cluster is added to the database. Otherwise, no accounting information can be written to the database. To add a cluster table, run sacctmgr -i add cluster CLUSTERNAME.

11. Restart slurmctld:

    management # systemctl restart slurmctld

12. (Optional) By default, Slurm does not take any group membership into account, and the system groups cannot be mapped to Slurm. Group creation and membership must be managed via the command line tool sacctmgr. You can have a group hierarchy, and users can be part of several groups.

    The following example creates an umbrella group bavaria for two subgroups called nuremberg and munich:

    management # sacctmgr add account bavaria \
    Description="umbrella group for subgroups" Organization=bavaria

    management # sacctmgr add account nuremberg,munich parent=bavaria \
    Description="subgroup" Organization=bavaria

    The following example adds a user called tux to the subgroup nuremberg:

    management # sacctmgr add user tux Account=nuremberg

1. Stop the slurmdbd service:

   DBnode # rcslurmdbd stop

   Ensure that slurmdbd is not running anymore:

   DBnode # rcslurmdbd status

   slurmctld might remain running while the database daemon is down. During this time, requests intended for slurmdbd are queued internally. The DBD Agent Queue size is limited, however, and should therefore be monitored with sdiag.

2. Create a backup of the slurm_acct_db database:

   DBnode # mysqldump -p slurm_acct_db > slurm_acct_db.sql

   If needed, this can be restored with the following command:

   DBnode # mysql -p slurm_acct_db < slurm_acct_db.sql

3. During the database conversion, the variable innodb_buffer_pool_size must be set to a value of 128 MB or more. Check the current size:

   DBnode # echo  'SELECT @@innodb_buffer_pool_size/1024/1024;' | \
   mysql --password --batch

4. If the value of innodb_buffer_pool_size is less than 128 MB, you can change it for the duration of the current session (on mariadb):

   DBnode # echo 'set GLOBAL innodb_buffer_pool_size = 134217728;' | \
   mysql --password --batch

   Alternatively, to permanently change the size, edit the /etc/my.cnf file, set innodb_buffer_pool_size to 128 MB, then restart the database:

   DBnode # rcmysql restart

5. If you need to update MariaDB, run the following command:

   DBnode # zypper update mariadb

   Convert the database tables to the new version of MariaDB:

   DBnode # mysql_upgrade --user=root --password=ROOT_DB_PASSWORD;

6. Install the new version of slurmdbd:

   DBnode # zypper install --force-resolution slurm_VERSION-slurmdbd

7. Rebuild the database. If you are using a backup slurmdbd, perform this step on the primary slurmdbd first.

   Because a conversion might take a considerable amount of time, the systemd service might time out during the conversion. Therefore, we recommend performing the migration manually by running slurmdbd from the command line in the foreground:

   DBnode # /usr/sbin/slurmdbd -D -v

   When you see the following message, you can shut down slurmdbd by pressing Ctrl–C:

   Conversion done:
   success!

8. Before restarting the service, remove or replace any deprecated configuration options. Check the deprecated options in the Release Notes.

9. Restart slurmdbd:

   DBnode # systemctl start slurmdbd

   

   Note: No daemonization during rebuild

   During the rebuild of the Slurm database, the database daemon does not daemonize.

1. Back up the configuration file/etc/slurm/slurm.conf. Because this file should be identical across the entire cluster, it is sufficient to do so only on the main management server.

2. On the main management server, edit /etc/slurm/slurm.conf and set SlurmdTimeout and SlurmctldTimeout to sufficiently high values to avoid timeouts while slurmctld and slurmd are down:

   SlurmctldTimeout=3600
   SlurmdTimeout=3600

   We recommend at least 60 minutes (3600), and more for larger clusters.

3. Copy the updated /etc/slurm/slurm.conf from the management server to all nodes:

   1. Obtain the list of partitions in /etc/slurm/slurm.conf.

   2. Copy the updated configuration to the compute nodes:

      management # cp /etc/slurm/slurm.conf /etc/slurm/slurm.conf.update
      management # sudo -u slurm /bin/bash -c 'cat /etc/slurm/slurm.conf.update | \
      pdsh -R mrsh -P PARTITIONS "cat > /etc/slurm/slurm.conf"'
      management # rm /etc/slurm/slurm.conf.update

   3. Reload the configuration file on all compute nodes:

      management # scontrol reconfigure

   4. Verify that the reconfiguration took effect:

      management # scontrol show config | grep Timeout

4. Shut down all running slurmctld instances, first on any backup management servers, and then on the main management server:

   management # systemctl stop slurmctld

5. Back up the slurmctld state files. slurmctld maintains persistent state information. Almost every major version involves changes to the slurmctld state files. This state information is upgraded if the upgrade remains within the supported version range and no data is lost.

   However, if a downgrade is necessary, state information from newer versions is not recognized by an older version of slurmctld and is discarded, resulting in a loss of all running and pending jobs. Therefore, back up the old state in case an update needs to be rolled back.

   1. Determine the StateSaveLocation directory:

      management # scontrol show config | grep StateSaveLocation

   2. Create a backup of the content of this directory. If a downgrade is required, restore the content of the StateSaveLocation directory from this backup.

6. Shut down slurmd on the compute nodes:

   management # pdsh -R ssh -P PARTITIONS systemctl stop slurmd

7. Upgrade slurmctld on the main and backup management servers:

   management # zypper install --force-resolution slurm_VERSION

   

   Important: Upgrade all Slurm packages at the same time

   If any additional Slurm packages are installed, you must upgrade those as well. This includes:

   • slurm-pam_slurm

   • slurm-sview

   • perl-slurm

   • slurm-lua

   • slurm-torque

   • slurm-config-man

   • slurm-doc

   • slurm-webdoc

   • slurm-auth-none

   • pdsh-slurm

   All Slurm packages must be upgraded at the same time to avoid conflicts between packages of different versions. This can be done by adding them to the zypper install command line described above.

8. Upgrade slurmd on the compute nodes:

   management # pdsh -R ssh -P PARTITIONS \
   zypper install --force-resolution slurm_VERSION-node

   

   Note: Memory size seen by slurmd might change on update

   Under certain circumstances, the amount of memory seen by slurmd might change after an update. If this happens, slurmctld puts the nodes in a drained state. To check whether the amount of memory seem by slurmd changed after the update, run the following command on a single compute node:

   node1 # slurmd -C

   Compare the output with the settings in slurm.conf. If required, correct the setting.

9. Before restarting the service, remove or replace any deprecated configuration options. Check the deprecated options in the Release Notes.

   If you replace deprecated options in the configuration files, these configuration files can be distributed to all management servers and compute nodes in the cluster by using the method described in Step 3.

10. Restart slurmd on all compute nodes:

    management # pdsh -R ssh -P PARTITIONS systemctl start slurmd

11. Restart slurmctld on the main and backup management servers:

    management # systemctl start slurmctld

12. Check the status of the management servers. On the main and backup management servers, run the following command:

    management # systemctl status slurmctld

13. Verify that the services are running without errors. Run the following command to check whether there are any down, drained, failing, or failed nodes:

    management # sinfo -R

14. Restore the original values of SlurmdTimeout and SlurmctldTimeout in /etc/slurm/slurm.conf, then copy the restored configuration to all nodes by using the method described in Step 3.

In this procedure, replace MNTRNODE with the host name or IP address of the server where Prometheus and Grafana are installed.

1. Install the Prometheus and Grafana packages:

   monitor # zypper in golang-github-prometheus-prometheus grafana

2. Enable and start Prometheus:

   monitor # systemctl enable --now prometheus

3. Verify that Prometheus works:

   • In a browser, navigate to MNTRNODE:9090/config, or:

   • In a terminal, run the following command:

     > wget MNTRNODE:9090/config --output-document=-

   Either of these methods should show the default contents of the /etc/prometheus/prometheus.yml file.

4. Enable and start Grafana:

   monitor # systemctl enable --now grafana-server

5. Log in to the Grafana web server at MNTRNODE:3000.

   Use admin for both the user name and password, then change the password when prompted.

6. On the left panel, select the gear icon () and click Data Sources.

7. Click Add data source.

8. Find Prometheus and click Select.

9. In the URL field, enter http://localhost:9090. The default settings for the other fields can remain unchanged.

   

   If Prometheus and Grafana are installed on different servers, replace localhost with the host name or IP address of the server where Prometheus is installed.

10. Click Save & Test.

In this procedure, replace MGMTSERVER with the host name or IP address of the server where the Slurm exporter is installed, and replace MNTRNODE with the host name or IP address of the server where Grafana is installed.

1. Install the Slurm exporter:

   management # zypper in golang-github-vpenso-prometheus_slurm_exporter

2. Enable and start the Slurm exporter:

   management # systemctl enable --now prometheus-slurm_exporter

   

   Important: Slurm exporter fails when GPU monitoring is enabled

   In Slurm 20.11, the Slurm exporter fails when GPU monitoring is enabled.

   This feature is disabled by default. Do not enable it for this version of Slurm.

3. Verify that the Slurm exporter works:

   • In a browser, navigate to MNGMTSERVER:8080/metrics, or:

   • In a terminal, run the following command:

     > wget MGMTSERVER:8080/metrics --output-document=-

   Either of these methods should show output similar to the following:

   # HELP go_gc_duration_seconds A summary of the GC invocation durations.
   # TYPE go_gc_duration_seconds summary
   go_gc_duration_seconds{quantile="0"} 1.9521e-05
   go_gc_duration_seconds{quantile="0.25"} 4.5717e-05
   go_gc_duration_seconds{quantile="0.5"} 7.8573e-05
   ...

4. On the server where Prometheus is installed, edit the scrape_configs section of the /etc/prometheus/prometheus.yml file to add a job for the Slurm exporter:

     - job_name: slurm-exporter
        scrape_interval: 30s
        scrape_timeout: 30s
        static_configs:
          - targets: ['MGMTSERVER:8080']

   Set the scrape_interval and scrape_timeout to 30s to avoid overloading the server.

5. Restart the Prometheus service:

   monitor # systemctl restart prometheus

6. Log in to the Grafana web server at MNTRNODE:3000.

7. On the left panel, select the plus icon () and click Import.

8. In the Import via grafana.com field, enter the dashboard ID 4323, then click Load.

9. From the Select a Prometheus data source drop-down box, select the Prometheus data source added in Procedure 6.1, “Installing Prometheus and Grafana”, then click Import.

10. Review the Slurm dashboard. The data might take some time to appear.

11. If you made any changes, click Save dashboard when prompted, optionally describe your changes, then click Save.

In this procedure, replace the example node names with the host names or IP addresses of the nodes, and replace MNTRNODE with the host name or IP address of the server where Grafana is installed.

1. Install the node exporter on each compute node. You can do this on multiple nodes at once by running the following command:

   management # pdsh -R ssh -u root -w "NODE1,NODE2" \
   "zypper in -y golang-github-prometheus-node_exporter"

2. Enable and start the node exporter. You can do this on multiple nodes at once by running the following command:

   management # pdsh -R ssh -u root -w "NODE1,NODE2" \
   "systemctl enable --now prometheus-node_exporter"

3. Verify that the node exporter works:

   • In a browser, navigate to NODE1:9100/metrics, or:

   • In a terminal, run the following command:

     > wget NODE1:9100/metrics --output-document=-

   Either of these methods should show output similar to the following:

   # HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
   # TYPE go_gc_duration_seconds summary
   go_gc_duration_seconds{quantile="0"} 2.3937e-05
   go_gc_duration_seconds{quantile="0.25"} 3.5456e-05
   go_gc_duration_seconds{quantile="0.5"} 8.1436e-05
   ...

4. On the server where Prometheus is installed, edit the scrape_configs section of the /etc/prometheus/prometheus.yml file to add a job for the node exporter:

     - job_name: node-exporter
       static_configs:
         - targets: ['NODE1:9100']
         - targets: ['NODE2:9100']

   Add a target for every node that has the node exporter installed.

5. Restart the Prometheus service:

   monitor # systemctl restart prometheus

6. Log in to the Grafana web server at MNTRNODE:3000.

7. On the left panel, select the plus icon () and click Import.

8. In the Import via grafana.com field, enter the dashboard ID 405, then click Load.

9. From the Select a Prometheus data source drop-down box, select the Prometheus data source added in Procedure 6.1, “Installing Prometheus and Grafana”, then click Import.

10. Review the node dashboard. Click the node drop-down box to select the nodes you want to view. The data might take some time to appear.

11. If you made any changes, click Save dashboard when prompted. To keep the currently selected nodes next time you access the dashboard, activate Save current variable values as dashboard default. Optionally describe your changes, then click Save.