Jump to content
SUSE Linux Enterprise High Performance Computing 15 SP3

Administration Guide

SUSE Linux Enterprise High Performance Computing

Publication Date: September 10, 2021

Copyright © 2020–2021 SUSE LLC and contributors. All rights reserved.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled GNU Free Documentation License.

For SUSE trademarks, see http://www.suse.com/company/legal/. All third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its affiliates. Asterisks (*) denote third-party trademarks.

All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its affiliates, the authors nor the translators shall be held liable for possible errors or the consequences thereof.

Preface

1 Available documentation

Online documentation

The online documentation for this product is available at https://documentation.suse.com/#sle-hpc. Browse or download the documentation in various formats.

Find the online documentation for other products at https://documentation.suse.com/.

Note
Note: Latest updates

The latest documentation updates are usually available in the English version of the documentation.

Release notes

For release notes, see https://www.suse.com/releasenotes/.

In your system

For offline use, find documentation in your installed system under /usr/share/doc. Many commands are also described in detail in their manual pages. To view them, run man, followed by a specific command name. If the man command is not installed on your system, install it with sudo zypper install man.

2 Giving feedback

We welcome feedback on, and contributions to, this documentation. There are several channels for this:

Service requests and support

For services and support options available for your product, see http://www.suse.com/support/.

To open a service request, you need a SUSE subscription registered at SUSE Customer Center. Go to https://scc.suse.com/support/requests, log in, and click Create New.

Bug reports

Report issues with the documentation at https://bugzilla.suse.com/. Reporting issues requires a Bugzilla account.

To simplify this process, you can use the Report Documentation Bug links next to headlines in the HTML version of this document. These preselect the right product and category in Bugzilla and add a link to the current section. You can start typing your bug report right away.

Contributions

To contribute to this documentation, use the Edit Source links next to headlines in the HTML version of this document. They take you to the source code on GitHub, where you can open a pull request. Contributing requires a GitHub account.

For more information about the documentation environment used for this documentation, see the repository's README at https://github.com/SUSE/doc-hpc.

Mail

You can also report errors and send feedback concerning the documentation to <>. Include the document title, the product version, and the publication date of the document. Additionally, include the relevant section number and title (or provide the URL) and provide a concise description of the problem.

3 Documentation conventions

The following notices and typographic conventions are used in this document:

  • /etc/passwd: Directory names and file names

  • PLACEHOLDER: Replace PLACEHOLDER with the actual value

  • PATH: An environment variable

  • ls, --help: Commands, options, and parameters

  • user: The name of user or group

  • package_name: The name of a software package

  • Alt, AltF1: A key to press or a key combination. Keys are shown in uppercase as on a keyboard.

  • File, File › Save As: menu items, buttons

  • AMD/Intel This paragraph is only relevant for the Intel 64/AMD64 architectures. The arrows mark the beginning and the end of the text block.

    IBM Z, POWER This paragraph is only relevant for the architectures IBM Z and POWER. The arrows mark the beginning and the end of the text block.

  • Chapter 1, Example chapter: A cross-reference to another chapter in this guide.

  • Commands that must be run with root privileges. Often you can also prefix these commands with the sudo command to run them as non-privileged user.

    root # command
    tux > sudo command
  • Commands that can be run by non-privileged users.

    tux > command
  • Notices

    Warning
    Warning: Warning notice

    Vital information you must be aware of before proceeding. Warns you about security issues, potential loss of data, damage to hardware, or physical hazards.

    Important
    Important: Important notice

    Important information you should be aware of before proceeding.

    Note
    Note: Note notice

    Additional information, for example about differences in software versions.

    Tip
    Tip: Tip notice

    Helpful information, like a guideline or a piece of practical advice.

  • Compact Notices

    Note

    Additional information, for example about differences in software versions.

    Tip

    Helpful information, like a guideline or a piece of practical advice.

4 Support

Find the support statement for SUSE Linux Enterprise High Performance Computing and general information about technology previews below. For details about the product lifecycle, see https://www.suse.com/lifecycle.

If you are entitled to support, find details on how to collect information for a support ticket at https://documentation.suse.com/sles-15/html/SLES-all/cha-adm-support.html.

4.1 Support statement for SUSE Linux Enterprise High Performance Computing

To receive support, you need an appropriate subscription with SUSE. To view the specific support offerings available to you, go to https://www.suse.com/support/ and select your product.

The support levels are defined as follows:

L1

Problem determination, which means technical support designed to provide compatibility information, usage support, ongoing maintenance, information gathering and basic troubleshooting using available documentation.

L2

Problem isolation, which means technical support designed to analyze data, reproduce customer problems, isolate problem area and provide a resolution for problems not resolved by Level 1 or prepare for Level 3.

L3

Problem resolution, which means technical support designed to resolve problems by engaging engineering to resolve product defects which have been identified by Level 2 Support.

For contracted customers and partners, SUSE Linux Enterprise High Performance Computing is delivered with L3 support for all packages, except for the following:

  • Technology previews.

  • Sound, graphics, fonts, and artwork.

  • Packages that require an additional customer contract.

  • Some packages shipped as part of the module Workstation Extension are L2-supported only.

  • Packages with names ending in -devel (containing header files and similar developer resources) will only be supported together with their main packages.

SUSE will only support the usage of original packages. That is, packages that are unchanged and not recompiled.

4.2 Technology previews

Technology previews are packages, stacks, or features delivered by SUSE to provide glimpses into upcoming innovations. Technology previews are included for your convenience to give you a chance to test new technologies within your environment. We would appreciate your feedback! If you test a technology preview, please contact your SUSE representative and let them know about your experience and use cases. Your input is helpful for future development.

Technology previews have the following limitations:

  • Technology previews are still in development. Therefore, they may be functionally incomplete, unstable, or in other ways not suitable for production use.

  • Technology previews are not supported.

  • Technology previews may only be available for specific hardware architectures.

  • Details and functionality of technology previews are subject to change. As a result, upgrading to subsequent releases of a technology preview may be impossible and require a fresh installation.

  • SUSE may discover that a preview does not meet customer or market needs, or does not comply with enterprise standards. Technology previews can be removed from a product at any time. SUSE does not commit to providing a supported version of such technologies in the future.

For an overview of technology previews shipped with your product, see the release notes at https://www.suse.com/releasenotes/x86_64/SLE-HPC/15-SP3.

1 Introduction

SUSE Linux Enterprise High Performance Computing is a highly scalable, high-performance parallel computing platform for modeling, simulation, and advanced analytics workloads.

1.1 Components provided

SUSE Linux Enterprise High Performance Computing 15 SP3 provides tools and libraries related to High Performance Computing. This includes:

  • A workload manager

  • Remote and parallel shells

  • Performance monitoring and measuring tools

  • A serial-console monitoring tool

  • A cluster power management tool

  • A tool for discovering the machine hardware topology

  • System monitoring

  • A tool for monitoring memory errors

  • A tool for determining the CPU model and its capabilities (x86-64 only)

  • A user-extensible heap manager capable of distinguishing between different kinds of memory (x86-64 only)

  • Various MPI implementations

  • Serial and parallel computational libraries providing common standards, such as BLAS, LAPACK, and others.

  • Serial and parallel libraries for the HDF5 file format

1.2 Hardware platform support

SUSE Linux Enterprise High Performance Computing 15 SP3 is available for the Intel 64/AMD64 (x86-64) and AArch64 platforms.

1.3 Support and life cycle

SUSE Linux Enterprise High Performance Computing 15 SP3 is supported throughout the life cycle of SUSE Linux Enterprise 15 SP3. Two life-cycle extensions, Extended Service Overlap Support (ESPOS) and Long Term Support Service (LTSS), are also available for this product. Any released package is fully maintained and supported until the availability of the next release.

For more information, see the Support Policy page at https://www.suse.com/support/policy.html.

1.4 Documentation and other information

  • Read the README files on the media.

  • Get detailed change log information about a particular package from the RPM (where FILENAME.rpm is the name of the RPM):

    rpm --changelog -qp FILENAME.rpm
  • Check the ChangeLog file in the top level of the media for a chronological log of all changes made to the updated packages.

  • The most recent version of the release notes is always available at https://www.suse.com/releasenotes.

  • The most recent version of this documentation is always available at https://documentation.suse.com/.

2 Installation and upgrade

SUSE Linux Enterprise High Performance Computing comes with preconfigured system roles for HPC. These roles provide a set of preselected packages typical for the specific role, and an installation workflow that configures the system to make the best use of system resources based on a typical use case for the role.

2.1 System roles for SUSE Linux Enterprise High Performance Computing 15 SP3

You can choose specific roles for the system based on modules selected during the installation process. When the HPC Module is enabled, the following roles are available:

HPC management server (head node)

This role includes the following features:

  • Uses Btrfs as the default root file system

  • Includes HPC-enabled libraries

  • Disables the firewall and Kdump services

  • Installs a controller for the Slurm workload manager

  • Mounts a large scratch partition to /var/tmp

HPC compute node

This role includes the following features:

  • Based on the minimal setup configuration

  • Uses XFS as the default root file system

  • Includes HPC-enabled libraries

  • Disables firewall and Kdump services

  • Installs a client for the Slurm workload manager

  • Does not create a separate /home partition

  • Mounts a large scratch partition to /var/tmp

HPC development node

This role includes the following features:

  • Includes HPC-enabled libraries

  • Adds compilers and development toolchains

The scratch partition /var/tmp/ is only created if there is sufficient space available on the installation medium (minimum 32 GB).

The Environment Module Lmod is installed for all roles. It is required at build time and runtime of the system. For more information, see Section 7.1, “Lmod — Lua-based environment modules”.

All libraries specifically built for HPC are installed under /usr/lib/hpc. They are not part of the standard search path, so the Lmod environment module system is required.

MUNGE authentication is installed for all roles. MUNGE keys are generated and must be copied to all nodes in the cluster. For more information, see Section 3.4, “MUNGE authentication”.

For the Ganglia monitoring system, the data collector ganglia-gmond is installed for every role, while the data aggregator ganglia-gmetad needs to be installed manually on the system which is expected to collect the data. For more information, see Section 6.2, “Ganglia — system monitoring”.

The system roles are only available for new installations of SUSE Linux Enterprise High Performance Computing.

2.2 Upgrading to SUSE Linux Enterprise High Performance Computing 15 SP3

You can upgrade to SLE HPC 15 SP3 from SLE HPC 15 SP2.

3 Remote administration

High Performance Computing clusters usually consist of a small set of identical compute nodes. However, large clusters could consist of thousands of machines. This chapter describes tools to help manage the compute nodes in a cluster.

3.1 Genders — static cluster configuration database

Genders is a static cluster configuration database used for configuration management. It allows grouping and addressing sets of nodes by attributes, and is used by a variety of tools. The Genders database is a text file that is usually replicated on each node in a cluster.

Perl, Python, Lua, C, and C++ bindings are supplied with Genders. Each package provides man pages or other documentation which describes the APIs.

3.1.1 Genders database format

The Genders database in SUSE Linux Enterprise High Performance Computing is a plain-text file called /etc/genders. It contains a list of node names with their attributes. Each line of the database can have one of the following formats.

nodename                attr[=value],attr[=value],...
nodename1,nodename2,... attr[=value],attr[=value],...
nodenames[A-B]          attr[=value],attr[=value],...

Node names are listed without their domain, and are followed by any number of spaces or tabs, then the comma-separated list of attributes. Every attribute can optionally have a value. The substitution string %n can be used in an attribute value to represent the node name. Node names can be listed on multiple lines, so a node's attributes can be specified on multiple lines. However, no single node can have duplicate attributes.

The attribute list must not contain spaces, and there is no provision for continuation lines. Commas and equals characters (=) are special, and cannot appear in attribute names or values. Comments are prefixed with the hash character (#) and can appear anywhere in the file.

Ranges for node names can be specified in the form prefix[a-c,n-p,...] as an alternative to explicit lists of node names. For example, node[01-03,06] would specify node01, node02, node03, and node06.

3.1.2 Nodeattr usage

The command line utility nodeattr can be used to query data in the genders file. When the genders file is replicated on all nodes, a query can be done without network access. The genders file can be called as follows:

# nodeattr [-q | -n | -s] [-r] attr[=val]

-q is the default option and prints a list of nodes with attr[=val].

The -c or -s options give a comma-separated or space-separated list of nodes with attr[=val].

If none of the formatting options are specified, nodeattr returns a zero value if the local node has the specified attribute, and non-zero otherwise. The -v option causes any value associated with the attribute to go to stdout. If a node name is specified before the attribute, the specified node is queried instead of the local node.

To print all attributes for a particular node, run the following command:

# nodeattr -l [node]

If no node parameter is given, all attributes of the local node are printed.

To perform a syntax check of the genders database, run the following command:

# nodeattr [-f genders] -k

To specify an alternative database location, use the option -f.

3.2 pdsh — parallel remote shell program

pdsh is a parallel remote shell that can be used with multiple back-ends for remote connections. It can run a command on multiple machines in parallel.

To install pdsh, run the command zypper in pdsh.

In SUSE Linux Enterprise High Performance Computing, the back-ends ssh, mrsh, and exec are supported. The ssh back-end is the default. Non-default login methods can be used by setting the PDSH_RCMD_TYPE environment variable, or by using the -R command argument.

When using the ssh back-end, you must use a non-interactive (passwordless) login method.

The mrsh back-end requires the mrshd daemon to be running on the client. The mrsh back-end does not require the use of reserved sockets, so it does not suffer from port exhaustion when running commands on many machines in parallel. For information about setting up the system to use this back-end, see Section 3.5, “mrsh/mrlogin — remote login using MUNGE authentication”.

Remote machines can be specified on the command line, or pdsh can use a machines file (/etc/pdsh/machines), dsh (Dancer's shell)-style groups or netgroups. It can also target nodes based on the currently running Slurm jobs.

The different ways to select target hosts are realized by modules. Some of these modules provide identical options to pdsh. The module loaded first will win and handle the option. Therefore, we recommended using a single method and specifying this with the -M option.

The machines file lists all target hosts, one per line. The appropriate netgroup can be selected with the -g command line option.

The following host-list plugins for pdsh are supported: machines, slurm, netgroup and dshgroup. Each host-list plugin is provided in a separate package. This avoids conflicts between command line options for different plugins which happen to be identical, and helps to keep installations small and free of unneeded dependencies. Package dependencies have been set to prevent the installation of plugins with conflicting command options. To install one of the plugins, run:

# zypper in pdsh-PLUGIN_NAME

For more information, see the man page pdsh.

3.3 PowerMan — centralized power control for clusters

PowerMan can control the following remote power control devices (RPC) from a central location:

  • local devices connected to a serial port

  • RPCs listening on a TCP socket

  • RPCs that are accessed through an external program

The communication to RPCs is controlled by expect-like scripts. For a list of currently supported devices, see the configuration file /etc/powerman/powerman.conf.

To install PowerMan, run zypper in powerman.

To configure PowerMan, include the appropriate device file for your RPC (/etc/powerman/*.dev) in /etc/powerman/powerman.conf and add devices and nodes. The device type needs to match the specification name in one of the included device files. The list of plugs used for nodes needs to match an entry in the plug name list.

After configuring PowerMan, start its service:

# systemctl start powerman.service

To start PowerMan automatically after every boot, run the following command:

# systemctl enable powerman.service

Optionally, PowerMan can connect to a remote PowerMan instance. To enable this, add the option listen to /etc/powerman/powerman.conf.

Important
Important: Unencrypted transfer

When connecting to a remote PowerMan instance, data is transferred unencrypted. Therefore, use this feature only if the network is appropriately secured.

3.4 MUNGE authentication

MUNGE allows for secure communications between different machines that share the same secret key. The most common use case is the Slurm workload manager, which uses MUNGE for the encryption of its messages. Another use case is authentication for the parallel shell mrsh.

3.4.1 Setting up MUNGE authentication

MUNGE uses UID/GID values to uniquely identify and authenticate users, so you must ensure that users who will authenticate across a network have matching UIDs and GIDs across all nodes.

MUNGE credentials have a limited time-to-live, so you must ensure that the time is synchronized across the entire cluster.

MUNGE is installed with the command zypper in munge. This also installs further required packages. A separate munge-devel package is available to build applications that require MUNGE authentication.

When installing the munge package, a new key is generated on every system. However, the entire cluster needs to use the same MUNGE key. Therefore, you must securely copy the MUNGE key from one system to all the other nodes in the cluster. You can accomplish this by using pdsh with SSH. Ensure that the key is only readable by the munge user (permissions mask 0400).

Procedure 3.1: Setting up MUNGE authentication
  1. On the server where MUNGE is installed, check the permissions, owner, and file type of the key file /etc/munge/munge.key:

    # stat --format "%F %a %G %U %n" /etc/munge/munge.key

    The settings should be as follows:

    400 regular file munge munge /etc/munge/munge.key
  2. Calculate the MD5 sum of munge.key:

    # md5sum /etc/munge/munge.key
  3. Copy the key to the listed nodes using pdcp:

    # pdcp -R ssh -w NODELIST /etc/munge/munge.key /etc/munge/munge.key
  4. Check the key settings on the remote nodes:

    # pdsh -R ssh -w HOSTLIST stat --format \"%F %a %G %U %n\" /etc/munge/munge.key
    # pdsh -R ssh -w HOSTLIST md5sum /etc/munge/munge.key

    Ensure that they match the settings on the MUNGE server.

3.4.2 Enabling and starting MUNGE

munged must be running on all nodes that use MUNGE authentication. If MUNGE is used for authentication across the network, it needs to run on each side of the communications link.

To start the service and ensure it is started after every reboot, run the following command on each node:

# systemctl enable --now munge.service

You can also use pdsh to run this command on multiple nodes at once.

3.5 mrsh/mrlogin — remote login using MUNGE authentication

mrsh is a set of remote shell programs using the MUNGE authentication system instead of reserved ports for security.

It can be used as a drop-in replacement for rsh and rlogin.

To install mrsh, do the following:

  • If only the mrsh client is required (without allowing remote login to this machine), use: zypper in mrsh.

  • To allow logging in to a machine, the server must be installed: zypper in mrsh-server.

  • To get a drop-in replacement for rsh and rlogin, run: zypper in mrsh-rsh-server-compat or zypper in mrsh-rsh-compat.

To set up a cluster of machines allowing remote login from each other, first follow the instructions for setting up and starting MUNGE authentication in Section 3.4, “MUNGE authentication”. After the MUNGE service successfully starts, enable and start mrlogin on each machine on which the user will log in:

# systemctl enable mrlogind.socket mrshd.socket
# systemctl start mrlogind.socket mrshd.socket

To start mrsh support at boot, run the following command:

# systemctl enable munge.service
# systemctl enable mrlogin.service

We do not recommend using mrsh when logged in as the user root. This is disabled by default. To enable it anyway, run the following command:

# echo "mrsh" >> /etc/securetty
# echo "mrlogin" >> /etc/securetty

4 Hardware

This chapter describes tools that can be used to obtain hardware infrastructure information for HPC applications.

4.1 cpuid

cpuid executes the x86 CPUID instruction and decodes and prints the results to stdout. Its knowledge of Intel, AMD and Cyrix CPUs is fairly complete. It specifically targets the Intel Xeon Phi architecture.

To install cpuid, run zypper in cpuid.

For information about runtime options for cpuid, see the man page cpuid(1).

Note that this tool is only available for x86-64.

4.2 hwloc — portable abstraction of hierarchical architectures for high-performance computing

hwloc provides CLI tools and a C API to obtain the hierarchical map of key computing elements, such as NUMA memory nodes, shared caches, processor packages, processor cores, processing units (logical processors or threads), and I/O devices. hwloc also gathers various attributes such as cache and memory information, and is portable across a variety of different operating systems and platforms. It can also assemble the topologies of multiple machines into a single one, so that applications can read the topology of an entire fabric or cluster at once.

lstopo allows the user to obtain the topology of a machine or convert topology information obtained on a remote machine into one of several output formats. In graphical mode (X11), it displays the topology in a window. Several other output formats are available as well, including plain text, PDF, PNG, SVG and FIG. For more information, see the man pages provided by hwloc and lstopo.

hwloc features full support for import and export of XML-formatted topology files via the libxml2 library.

The package hwloc-devel offers a library that can be directly included into external programs. This requires that the libxml2 development library (package libxml2-devel) is available when compiling hwloc.

5 Slurm — utility for HPC workload management

Slurm is a workload manager for managing compute jobs on High Performance Computing clusters. It can start multiple jobs on a single node, or a single job on multiple nodes. Additional components can be used for advanced scheduling and accounting.

The mandatory components of Slurm are the control daemon slurmctld, which handles job scheduling, and the Slurm daemon slurmd, responsible for launching compute jobs. Nodes running slurmctld are called control nodes and nodes running slurmd are called compute nodes.

Additional components are a secondary slurmctld acting as a standby server for a failover, and the Slurm database daemon slurmdbd, which stores the job history and user hierarchy.

5.1 Installing Slurm

These instructions describe a minimal installation of Slurm with one control node and multiple compute nodes.

5.1.1 Minimal installation

Important
Important: Make sure of consistent UIDs and GIDs for Slurm's accounts

For security reasons, Slurm does not run as the user root, but under its own user. It is important that the user slurm has the same UID/GID across all nodes of the cluster.

If this user/group does not exist, the package slurm creates this user and group when it is installed. However, this does not guarantee that the generated UIDs/GIDs will be identical on all systems.

Therefore, we strongly advise you to create the user/group slurm before installing slurm. If you are using a network directory service such as LDAP for user and group management, you can use it to provide the slurm user/group as well.

It is strongly recommended that all compute nodes share common user home directories. These should be provided through network storage.

Procedure 5.1: Installing the Slurm packages
  1. On the control node, install the slurm package with the command zypper in slurm.

  2. On the compute nodes, install the slurm-node package with the command zypper in slurm-node.

  3. On the control node and the compute nodes, the package munge is installed automatically. Configure, enable and start MUNGE on the control and compute nodes as described in Section 3.4, “MUNGE authentication”. Ensure that the same munge key is shared across all nodes.

Procedure 5.2: Configuring Slurm
  1. On the control node, edit the main configuration file /etc/slurm/slurm.conf:

    1. Configure the parameter ControlMachine=CONTROL_MACHINE with the host name of the control node.

      To find the correct host name, run hostname -s on the control node.

    2. Under the COMPUTE NODES section, add the following lines to define the compute nodes:

      NodeName=NODE_LIST State=UNKNOWN
      PartitionName=normal Nodes=NODE_LIST Default=YES MaxTime=24:00:00 State=UP

      Replace NODE_LIST with the host names of the compute nodes, either comma-separated or as a range (for example: node[1-100]).

      The NodeName line also allows specifying additional parameters for the nodes, such as Boards, SocketsPerBoard CoresPerSocket, ThreadsPerCore, or CPU. The actual values of these can be obtained by running the following command on the compute nodes:

      # slurmd -C
  2. Copy the modified configuration file /etc/slurm/slurm.conf from the control node to all compute nodes:

    # scp /etc/slurm/slurm.conf COMPUTE_NODE:/etc/slurm/
  3. On the control node, start slurmctld and enable it so that it starts on every boot:

    # systemctl enable --now slurmctld.service
  4. On the compute nodes, start slurmd and enable it so that it starts on every boot:

    # systemctl enable --now slurmd.service
Procedure 5.3: Testing the Slurm installation
  1. Check the status and availability of the compute nodes by running the sinfo command. You should see output similar to the following:

    PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
    normal*      up 1-00:00:00      2   idle node[01-02]

    If the node state is not idle, see Section 5.4.1, “Frequently asked questions”.

  2. Test the Slurm installation by running the following command:

    # srun sleep 30

    This runs the sleep command on a free compute node for 30 seconds.

    In another shell, run the squeue command during the 30 seconds that the compute node is asleep. You should see output similar to the following:

    JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
        1    normal    sleep     root  R       0:05      1 node02
  3. Create the following shell script and save it as sleeper.sh:

    #!/bin/bash
    echo "started at $(date)"
    sleep 30
    echo "finished at $(date)"

    Run the shell script in the queue:

    # sbatch sleeper.sh

    The shell script is executed when enough resources are available, and the output is stored in the file slurm-${JOBNR}.out.

5.1.2 Installing the Slurm database

In a minimal installation, Slurm only stores pending and running jobs. To store finished and failed job data, the storage plugin must be installed and enabled. You can also enable completely fair scheduling, which replaces FIFO (first in, first out) scheduling with algorithms that calculate the job priority in a queue in dependence of the job which a user has run in the history.

The Slurm database has two components: the slurmdbd daemon itself, and an SQL database. MariaDB is recommended. The database can be installed on the same node that runs slurmdbd, or on a separate node. For a minimal setup, all these services run on the same node.

Procedure 5.4: Install slurmdbd
Note
Note: MariaDB

If you want to use an external SQL database (or you already have a database installed on the control node), you can skip the MariaDB steps.

  1. Install the MariaDB SQL database with zypper in mariadb.

  2. Start and enable MariaDB:

    # systemctl enable --now mariadb
  3. Secure the database with the command mysql_secure_installation.

  4. Connect to the SQL database, for example with the command mysql -u root -p.

  5. After a successful connection to the database and the creation of a secure password [1] , create the Slurm user and the database with the following commands:

    create user 'slurm'@'localhost' identified by 'password';
    grant all on slurm_acct_db.* TO 'slurm'@'localhost';
    create database slurm_acct_db;

    After these steps are complete, exit the database.

  6. Install the slurmdbd package:

    # zypper in slurm-slurmdbd
  7. Edit the /etc/slurm/slurmdbd.conf file so that the daemon can access the database. Change the following line to the password that you used in Step 5:

    StoragePass=password

    If you chose another location or user for the SQL database, you must also modify the following lines:

    StorageUser=slurm
    DbdAddr=localhost
    DbdHost=localhost
  8. Start and enable slurmdbd:

    # systemctl enable --now slurmdbd

    The first start of slurmdbd will take some time.

  9. To enable accounting, edit the /etc/slurm/slurm.conf file to add the connection between slurmctld and the slurmdbd daemon. Ensure that the following lines appear as shown:

    JobAcctGatherType=jobacct_gather/linux
    JobAcctGatherFrequency=30
    AccountingStorageType=accounting_storage/slurmdbd
    AccountingStorageHost=localhost
    Note

    This example assumes that slurmdbd is running on the same node as slurmctld. If not, change localhost to the host name or IP address of the node where slurmdbd is running.

  10. Ensure that a table for the cluster is added to the database. Otherwise, no accounting information can be written to the database. To add a cluster table, run sacctmgr -i add cluster CLUSTERNAME.

  11. Restart slurmctld:

    # systemctl restart slurmctld
  12. (Optional) By default, Slurm does not take any group membership into account, and the system groups cannot be mapped to Slurm. Group creation and membership must be managed via the command line tool sacctmgr. You can have a group hierarchy, and users can be part of several groups.

    The following example creates an umbrella group bavaria for two subgroups called nuremberg and munich:

    # sacctmgr add account bavaria \
    Description="umbrella group for subgroups" Organization=bavaria
    # sacctmgr add account nuremberg,munich parent=bavaria \
    Description="subgroup" Organization=bavaria

    The following example adds a user called tux to the subgroup nuremberg:

    # sacctmgr add user tux Account=nuremberg

5.2 Slurm administration commands

5.2.1 scontrol

The command scontrol is used to show and update the entities of Slurm, such as the state of the compute nodes or compute jobs. It can also be used to reboot or to propagate configuration changes to the compute nodes.

Useful options for this command are --details, which prints more verbose output, and --oneliner, which forces the output onto a single line, which is more useful in shell scripts.

scontrol show ENTITY

Displays the state of the specified ENTITY.

scontrol update SPECIFICATION

Updates the SPECIFICATION like the compute node or compute node state. Useful SPECIFICATION states that can be set for compute nodes include:

nodename=NODE state=down reason=REASON

Removes all jobs from the compute node, and aborts any jobs already running on the node.

nodename=NODE state=drain reason=REASON

Drains the compute node so that no new jobs can be scheduled on it, but does not end compute jobs already running on the compute node. REASON can be any string. The compute node stays in the drained state and must be returned to the idle state manually.

nodename=NODE state=resume

Marks the compute node as ready to return to the idle state.

jobid=JOBID REQUIREMENT=VALUE

Updates the given requirement, such as NumNodes, with a new value. This command can also be run as a non-privileged user.

scontrol reconfigure

Triggers a reload of the configuration file slurm.conf on all compute nodes.

scontrol reboot NODELIST

Reboots a compute node, or group of compute nodes, when the jobs on it finish. To use this command, the option RebootProgram="/sbin/reboot" must be set in slurm.conf. When the reboot of a compute node takes more than 60 seconds, you can set a higher value in slurm.conf, such as ResumeTimeout=300.

sacctmgr

Used for job accounting in Slurm. To use this command, the service slurmdbd must be set up. See Section 5.1.2, “Installing the Slurm database”.

sinfo

Retrieves information about the state of the compute nodes, and can be used for a fast overview of the cluster health. The following command line switches are available:

--dead

Displays information about unresponsive nodes.

--long

Shows more detailed information.

--reservation

Prints information about advanced reservations.

-R

Displays the reason a node is in the down, drained, or failing state.

--state=STATE

Limits the output only to nodes with the specified STATE.

sacct

Displays the accounting data if accounting is enabled. The following options are available:

--allusers

Shows accounting data for all users.

--accounts=NAME

Shows only the specified user(s).

--starttime=MM/DD[/YY]-HH:MM[:SS]

Shows only jobs after the specified start time. You can use just MM/DD or HH:MM. If no time is given, the command defaults to 00:00, which means that only jobs from today are shown.

--endtime=MM/DD[/YY]-HH:MM[:SS]

Accepts the same options as --starttime. If no time is given, the time when the command was issued is used.

--name=NAME

Limits output to jobs with the given NAME.

--partition=PARTITION

Shows only jobs that run in the specified PARTITION.

sbatch, salloc and srun

These commands are used to schedule compute jobs, which means batch scripts for the sbatch command, interactive sessions for the salloc command, or binaries for the srun command. If the job cannot be scheduled immediately, only sbatch places it into the queue.

Frequently-used options for these commands include:

-n COUNT_THREADS

Specifies the number of threads needed by the job. The threads can be allocated on different nodes.

-N MINCOUNT_NODES[-MAXCOUNT_NODES]

Sets the number of compute nodes required for a job. The MAXCOUNT_NODES number can be omitted.

--time TIME

Specifies the maximum clock time (runtime) after which a job is terminated. The format of TIME is either seconds or [HH:]MM:SS. Not to be confused with walltime, which is clocktime × threads.

--signal [B:]NUMBER[@TIME]

Sends the signal specified by NUMBER 60 seconds before the end of the job, unless TIME is specified. The signal is sent to every process on every node. If a signal should only be sent to the controlling batch job, you must specify the B: flag.

--job-name NAME

Sets the name of the job to NAME in the queue.

--array=RANGEINDEX

Executes the given script via sbatch for indexes given by RANGEINDEX with the same parameters.

--dependency=STATE:JOBID

Defers the job until the specified STATE of the job JOBID is reached.

--gres=GRES

Runs a job only on nodes with the specified generic resource (GRes), for example a GPU, specified by the value of GRES.

--licenses=NAME[:COUNT]

The job must have the specified number (COUNT) of licenses with the name NAME. A license is the opposite of a generic resource: it is not tied to a computer, but is a cluster-wide variable.

--mem=MEMORY

Sets the real MEMORY required by a job per node. To use this option, memory control must be enabled. The default unit for the MEMORY value is megabytes, but you can also use K for kilobyte, M for megabyte, G for gigabyte, or T for terabyte.

--mem-per-cpu=MEMORY

This option takes the same values as --mem, but defines memory on a per-CPU basis rather than a per-node basis.

5.3 Upgrading Slurm

For existing products under general support, version upgrades of Slurm are provided regularly. With some restrictions, interoperability is guaranteed between three consecutive versions.

Unlike maintenance updates, these upgrades are not installed automatically using zypper patch but require the administrator to request their installation explicitly. This ensures that these upgrades are not installed unintentionally and gives the administrator the opportunity to plan version upgrades beforehand.

In addition to the three major-version rule, obey the following rules regarding the order of updates:

  1. The version of slurmdbd must be identical to or higher than the version of slurmctld.

  2. The version of slurmctld must the identical to or higher than the version of slurmd.

  3. The version of slurmd must be identical to or higher than the version of the slurm user applications.

Or in short: version(slurmdbd) >= version(slurmctld) >= version(slurmd) >= version (Slurm user CLIs).

With each version, configuration options for slurmctld/slurmd or slurmdbd might be deprecated. While deprecated, they remain valid for this version and the two consecutive versions, but they might be removed later. Therefore, it is advisable to update the configuration files after the upgrade and replace deprecated configuration options before the final restart of a service.

Slurm uses a segmented version number: the first two segments denote the major version, and the final segment denotes the patch level. Upgrade packages (that is, packages that were not initially supplied with the module or service pack) have their major version encoded in the package name (with periods . replaced by underscores _). For example, for version 18.08, this would be slurm_18_08-*.rpm.

To upgrade the package slurm to 18.08, run the following command:

zypper install --force-resolution slurm_18_08

To upgrade Slurm subpackages, use the analogous commands.

Important
Important

If any additional Slurm packages are installed, you must upgrade those as well. This includes:

  • slurm-pam_slurm

  • slurm-sview

  • perl-slurm

  • slurm-lua

  • slurm-torque

  • slurm-config-man

  • slurm-doc

  • slurm-webdoc

  • slurm-auth-none

  • pdsh-slurm

All Slurm packages must be upgraded at the same time to avoid conflicts between packages of different versions. This can be done by adding them to the zypper install command line described above.

Note
Note

A new major version of Slurm introduces a new version of libslurm. Older versions of this library might not work with an upgraded Slurm. An upgrade is provided for all SUSE Linux Enterprise software that depends on libslurm . Any locally-developed Slurm modules or tools might require modification or recompilation.

5.3.1 Upgrade workflow

This workflow assumes that MUNGE authentication is used and that pdsh, the pdsh Slurm plugin, and mrsh can access all of the machines in the cluster (that is, mrshd is running on all nodes in the cluster).

If this is not the case, install pdsh:

# zypper in pdsh-slurm

If mrsh is not used in the cluster, the ssh back-end for pdsh can be used instead. Replace the option -R mrsh with -R ssh in the pdshcommands below. This is less scalable and you might run out of usable ports.

Warning
Warning: Upgrade slurmdbd databases before other Slurm components

If slurmdbd is used, always upgrade the slurmdbd database before starting the upgrade of any other Slurm component. The same database can be connected to multiple clusters and must be upgraded before all of them.

Upgrading other Slurm components before the database can lead to data loss.

Procedure 5.5: Upgrading Slurm
  1. Upgrade the slurmdbd database daemon

    When upgrading slurmdbd, the database is converted when the new version of slurmdbd starts for the first time. If the database is big, the conversion could take several tens of minutes. During this time, the database is inaccessible.

    It is highly recommended to create a backup of the database in case an error occurs during or after the upgrade process. Without a backup, all accounting data collected in the database might be lost if an error occurs or the upgrade is rolled back. A database converted to a newer version cannot be converted back to an older version, and older versions of slurmdbd do not recognize the newer formats. To back up and upgrade slurmdbd, follow this procedure:

    1. Stop the slurmdbd service:

      # rcslurmdbd stop

      Ensure that slurmdbd is not running anymore:

      # rcslurmdbd status

      slurmctld might remain running while the database daemon is down. During this time, requests intended for slurmdbd are queued internally. The DBD Agent Queue size is limited, however, and should therefore be monitored with sdiag.

    2. Create a backup of the slurm_acct_db database:

      # mysqldump -p slurm_acct_db > slurm_acct_db.sql

      If needed, this can be restored with the following command:

      # mysql -p slurm_acct_db < slurm_acct_db.sql
    3. In preparation for the conversion, ensure that the variable innodb_buffer_pool_size is set to a value of 128 Mb or more:

      On the database server, run the following command:

      # echo  'SELECT @@innodb_buffer_pool_size/1024/1024;' | \
      mysql --password --batch

      If the size is less than 128 MB, you can change it for the duration of the current session (on mariadb):

      # echo 'set GLOBAL innodb_buffer_pool_size = 134217728;' | \
      mysql --password --batch

      To permanently change the size, edit the /etc/my.cnf file, set innodb_buffer_pool_size to 128 MB, then restart the database:

      # rcmysql restart
    4. If you also need to update mariadb, run the following command:

      # zypper update mariadb

      Convert the database tables to the new version of MariaDB:

      # mysql_upgrade --user=root --password=root_db_password;
    5. Install the new version of slurmdbd:

      # zypper install --force-resolution slurm_version-slurmdbd
    6. Rebuild database

      Because a conversion might take a considerable amount of time, the systemd service might time out during the conversion. Therefore, we recommend performing the migration manually by running slurmdbd from the command line in the foreground:

      # /usr/sbin/slurmdbd -D -v

      Once you see the following message, you can shut down slurmdbd by pressing CtrlC:

      Conversion done:
      success!
      Note
      Note: Convert primary slurmdbd first

      If you are using a backup slurmdbd, the conversion must be performed on the primary slurmdbd first. The backup slurmdbd only starts after the conversion is complete.

    7. Before restarting the service, remove or replace any deprecated configuration options. Check the deprecated options in the Release Notes. After this is complete, restart slurmdbd:

      # systemctl start slurmdbd
      Note
      Note: No daemonization during rebuild

      During the rebuild of the Slurm database, the database daemon does not daemonize.

  2. Update slurmctld and slurmd

    After the Slurm database is updated, the slurmctld and slurmd instances can be updated. It is recommended to update the control and compute nodes all in a single pass. If this is not feasible, the compute nodes (slurmd) can be updated on a node-by-node basis. However, the control nodes (slurmctld) must be updated first.

    1. Back up the slurmctld and slurmd configuration

      Create a backup copy of the Slurm configuration before starting the upgrade process. Since the configuration file /etc/slurm/slurm.conf should be identical across the entire cluster, it is sufficient to do so only on the main control node.

    2. Increase Timeouts

      Edit /etc/slurm/slurm.conf and set SlurmdTimeout and SlurmctldTimeout to sufficiently high values to avoid timeouts while slurmctld and slurmd are down. We recommend at least 60 minutes, and more on larger clusters.

      1. On the main control node, edit /etc/slurm/slurm.conf and set the values for the following variables to at least 3600 (one hour):

        SlurmctldTimeout=3600
        SlurmdTimeout=3600
      2. Copy /etc/slurm/slurm.conf to all nodes. To do so using the following steps, MUNGE authentication must be used in the cluster as recommended.

        1. Obtain the list of partitions in /etc/slurm/slurm.conf.

        2. Run the following commands:

          # cp /etc/slurm/slurm.conf /etc/slurm/slurm.conf.update
          # sudo -u slurm /bin/bash -c 'cat /etc/slurm/slurm.conf.update | \
          pdsh -R mrsh -P partitions \
          "cat > /etc/slurm/slurm.conf"'
          # rm /etc/slurm/slurm.conf.update
          # scontrol reconfigure
        3. Verify that the reconfiguration took effect:

          # scontrol show config | grep Timeout
    3. Shut down any running slurmctld instances

      1. If applicable, shut down any backup controllers on the backup control nodes:

        backup: # systemctl stop slurmctld
      2. Shut down the main controller:

        master: # systemctl stop slurmctld
    4. Back up the slurmctld state files

      slurmctld maintains persistent state information. Almost every major version involves changes to the slurmctld state files. This state information is upgraded if the upgrade remains within the supported version range and no data is lost.

      However, if a downgrade is necessary, state information from newer versions is not recognized by an older version of slurmctld and is discarded, resulting in a loss of all running and pending jobs. Therefore, back up the old state in case an update needs to be rolled back.

      1. Determine the StateSaveLocation directory:

        # scontrol show config | grep StateSaveLocation
      2. Create a backup of the content of this directory.

        If a downgrade is required, restore the content of the StateSaveLocation directory from this backup.

    5. Shut down slurmd on the nodes

      # pdsh -R ssh -P partitions systemctl stop slurmd
    6. Upgrade slurmctld on the control and backup nodes as well as slurmd on the compute nodes

      1. On the main and backup control nodes, run the following command:

        master: # zypper zypper install \
        --force-resolution slurm_version
      2. On the control node, run the following command:

        master: # pdsh -R ssh -P partitions \
        zypper install --force-resolution \
        slurm_version-node
      Note
      Note: Memory size seen by slurmd might change on update

      Under certain circumstances, the amount of memory seen by slurmd might change after an update. If this happens, slurmctld puts the nodes in a drained state. To check whether the amount of memory seem by slurmd changed after the update, run the following command on a single compute node:

      node1: # slurmd -C

      Compare the output with the settings in slurm.conf. If required, correct the setting.

    7. Replace deprecated options

      If you replace deprecated options in the configuration files, these configuration files can be distributed to all controllers and nodes in the cluster by using the method described in Step 2.b.ii.B.

    8. Restart slurmd on all compute nodes

      On the main control node, run the following command:

      master: # pdsh -R ssh -P partitions \
      systemctl start slurmd

      On the main and backups control nodes, run the following command:

      master: # systemctl start slurmctld
    9. Verify that the system operates properly

      1. Check the status of the control nodes. On the main and backup control nodes, run the following command:

        # systemctl status slurmctld
      2. Verify that the services are running without errors. Run the following command to check whether there are any down, drained, failing, or failed nodes:

        # sinfo -R
    10. Clean up

      Restore the SlurmdTimeout and SlurmctldTimeout values in /etc/slurm/slurm.conf on all nodes, then run scontrol reconfigure (see Step 2.b).

A new major version of libslurm is provided with each service pack of SUSE Linux Enterprise High Performance Computing. The old version is not uninstalled on upgrade, and user-provided applications built with an old version should continue to work if the old library used is not older than the past two versions. It is strongly recommended to rebuild local applications using libslurm — such as MPI libraries with Slurm support — as early as possible. This might require updating the user applications, as new arguments might be introduced to existing functions.

5.4 Additional resources

5.4.1 Frequently asked questions

Q: 1. How do I change the state of a node from down to up?

When the slurmd daemon on a node does not reboot in the time specified in the ResumeTimeout parameter, or the ReturnToService was not changed in the configuration file slurm.conf, compute nodes stay in the down state and must be set back to the up state manually. This can be done for the NODE with the following command:

# scontrol update state=resume NodeName=NODE
Q: 2. What is the difference between the states down and down*?

A * shown after a status code means that the node is not responding.

When a node is marked as down*, it means that the node is not reachable because of network issues, or that slurmd is not running on that node.

In the down state, the node is reachable, but either the node was rebooted unexpectedly, the hardware does not match the description in slurm.conf, or a health check was configured with the HealthCheckProgram.

Q: 3. How do I get the exact core count, socket number, and number of CPUs for a node?

To find the node values that go into the configuration file slurm.conf, run the following command:

# slurmd -C

5.4.2 External documentation

For further documentation, see the Quick Start Administrator Guide and Quick Start User Guide. There is further in-depth documentation on the Slurm documentation page.



[1] You can use the command openssl rand -base64 32 to create a secure random password

6 Monitoring and logging

Obtaining and maintaining an overview over the status and health of a cluster's compute nodes helps to ensure a smooth operation. This chapter describes tools that give an administrator an overview of the current cluster status, collect system logs, and gather information on certain system failure conditions.

6.1 ConMan — the console manager

ConMan is a serial console management program designed to support many console devices and simultaneous users. It supports:

  • local serial devices

  • remote terminal servers (via the telnet protocol)

  • IPMI Serial-Over-LAN (via FreeIPMI)

  • Unix domain sockets

  • external processes (for example, using expect scripts for telnet, ssh, or ipmi-sol connections)

ConMan can be used for monitoring, logging, and optionally timestamping console device output.

To install ConMan, run zypper in conman.

Important
Important: conmand sends unencrypted data

The daemon conmand sends unencrypted data over the network and its connections are not authenticated. Therefore, it should be used locally only, listening to the port localhost. However, the IPMI console does offer encryption. This makes conman a good tool for monitoring many such consoles.

ConMan provides expect-scripts in the directory /usr/lib/conman/exec.

Input to conman is not echoed in interactive mode. This can be changed by entering the escape sequence &E.

When pressing Enter in interactive mode, no line feed is generated. To generate a line feed, press CtrlL.

For more information about options, see the ConMan man page.

6.2 Ganglia — system monitoring

Ganglia is a scalable, distributed monitoring system for high-performance computing systems, such as clusters and grids. It is based on a hierarchical design targeted at federations of clusters.

6.2.1 Using Ganglia

To use Ganglia, install ganglia-gmetad on the management server, then start the Ganglia meta-daemon: rcgmead start. To make sure the service is started after a reboot, run: systemctl enable gmetad. On each cluster node which you want to monitor, install ganglia-gmond, start the service rcgmond start and make sure it is enabled to start automatically after a reboot: systemctl enable gmond. To test whether the gmond daemon has connected to the meta-daemon, run gstat -a and check that each node to be monitored is present in the output.

6.2.2 Ganglia on Btrfs

When using the Btrfs file system, the monitoring data will be lost after a rollback of the service gmetad. To fix this issue, either install the package ganglia-gmetad-skip-bcheck or create the file /etc/ganglia/no_btrfs_check.

6.2.3 Using the Ganglia Web interface

Install ganglia-web on the management server. Enable PHP in Apache2: a2enmod php7. Then start Apache2 on this machine: rcapache2 start and make sure it is started automatically after a reboot: systemctl enable apache2. The Ganglia Web interface is accessible from http://MANAGEMENT_SERVER/ganglia.

6.3 rasdaemon — utility to log RAS error tracings

rasdaemon is an RAS (Reliability, Availability and Serviceability) logging tool. It records memory errors using EDAC (Error Detection and Correction) tracing events. EDAC drivers in the Linux kernel handle detection of ECC (Error Correction Code) errors from memory controllers.

rasdaemon can be used on large memory systems to track, record, and localize memory errors and how they evolve over time to detect hardware degradation. Furthermore, it can be used to localize a faulty DIMM on the mainboard.

To check whether the EDAC drivers are loaded, run the following command:

# ras-mc-ctl --status

The command should return ras-mc-ctl: drivers are loaded. If it indicates that the drivers are not loaded, EDAC may not be supported on your board.

To start rasdaemon, run systemctl start rasdaemon.service. To start rasdaemon automatically at boot time, run systemctl enable rasdaemon.service. The daemon logs information to /var/log/messages and to an internal database. A summary of the stored errors can be obtained with the following command:

# ras-mc-ctl --summary

The errors stored in the database can be viewed with:

# ras-mc-ctl --errors

Optionally, you can load the DIMM labels silk-screened on the system board to more easily identify the faulty DIMM. To do so, before starting rasdaemon, run:

# systemctl start ras-mc-ctl start

For this to work, you need to set up a layout description for the board. There are no descriptions supplied by default. To add a layout description, create a file with an arbitrary name in the directory /etc/ras/dimm_labels.d/. The format is:

Vendor: MOTHERBOARD-VENDOR-NAME
Model: MOTHERBOARD-MODEL-NAME
  LABEL: MC.TOP.MID.LOW

7 HPC user libraries

Many HPC clusters need to accommodate multiple compute applications, each of which has its own very specific library dependencies. Multiple instances of the same libraries might exist, differing in version, build configuration, compiler, and MPI implementation. To manage these dependencies, you can use an Environment Module system. Most HPC libraries provided with SUSE Linux Enterprise High Performance Computing are built with support for environment modules. This chapter describes the Environment Module system Lmod, and a set of HPC compute libraries shipped with SLE HPC.

7.1 Lmod — Lua-based environment modules

Lmod is an advanced environment module system that allows the installation of multiple versions of a program or shared library, and helps configure the system environment for the use of a specific version. It supports hierarchical library dependencies and makes sure that the correct versions of dependent libraries are selected. Environment Modules-enabled library packages supplied with the HPC module support parallel installation of different versions and flavors of the same library or binary and are supplied with appropriate lmod module files.

7.1.1 Installation and basic usage

To install Lmod, run zypper in lua-lmod.

Before you can use Lmod, you must source an init file into the initialization file of your interactive shell. The following init files are available for various common shells:

/usr/share/lmod/lmod/init/bash
/usr/share/lmod/lmod/init/ksh
/usr/share/lmod/lmod/init/tcsh
/usr/share/lmod/lmod/init/zsh
/usr/share/lmod/lmod/init/sh

Pick the appropriate file for your shell, then add the following line into your shell's init file:

source /usr/share/lmod/lmod/init/INIT-FILE

The init script adds the command module.

7.1.2 Listing available modules

To list all the available modules, run module spider. To show all modules which can be loaded with the currently loaded modules, run module avail. A module name consists of a name and a version string, separated by a / character. If more than one version is available for a certain module name, the default version is marked by a * character. If there is no default, the module with the highest version number is loaded. To reference a specific module version, you can use the full string NAME/VERSION.

7.1.3 Listing loaded modules

module list shows all currently loaded modules. Refer to module help for some short help on the module command, and module help MODULE-NAME for help on the particular module. The module command is only available when you log in after installing lua-lmod.

7.1.4 Gathering information about a module

To get information about a particular module, run module whatis MODULE-NAME. To load a module, run module load MODULE-NAME. This will ensure that your environment is modified (that is, the PATH and LD_LIBRARY_PATH and other environment variables are prepended) so that binaries and libraries provided by the respective modules are found. To run a program compiled against this library, the appropriate module load commands must be issued beforehand.

7.1.5 Loading modules

The module load MODULE command must be run in the shell from which the module is to be used. Some modules require a compiler toolchain or MPI flavor module to be loaded before they are available for loading.

7.1.6 Environment variables

If the respective development packages are installed, build-time environment variables like LIBRARY_PATH, CPATH, C_INCLUDE_PATH, and CPLUS_INCLUDE_PATH are set up to include the directories containing the appropriate header and library files. However, some compiler and linker commands might not honor these. In this case, use the appropriate options together with the environment variables -I PACKAGE_NAME_INC and -L PACKAGE_NAME_LIB to add the include and library paths to the command lines of the compiler and linker.

7.1.7 For more information

For more information on Lmod, see https://lmod.readthedocs.org.

7.2 GNU Compiler Toolchain Collection for HPC

In SUSE Linux Enterprise High Performance Computing, the GNU compiler collection version 7 is provided as the base compiler toolchain. The gnu-compilers-hpc package provides the environment module for the base version of the GNU compiler suite. This package must be installed when using any of the HPC libraries enabled for environment modules.

7.2.1 Environment module

This package requires lua-lmod to supply environment module support.

To install gnu-compilers-hpc, run the following command:

# zypper in gnu-compilers-hpc

To make libraries built with the base compilers available, you must set up the environment appropriately and select the GNU toolchain. To do so, run the following command:

# module load gnu

7.2.2 Building High Performance Computing software with GNU Compiler Suite

To use the GNU compiler collection to build your own libraries and applications, gnu-compilers-hpc-devel must be installed. It ensures that all compiler components required for HPC (that is, C, C++, and Fortran compilers) are installed.

The environment variables CC, CXX, FC and F77 will be set correctly and the path will be adjusted so that the correct compiler version can be found.

7.2.3 Later versions

The Development Tools Module might provide later versions of the GNU compiler suite. To determine the available compiler suites, run the following command:

# zypper search '*-compilers-hpc'

If you have more than one version of the compiler suite installed, Lmod picks the latest one by default. If you require an older version, or the base version, append the version number:

# module load gnu/7

For more information, see Section 7.1, “Lmod — Lua-based environment modules”.

7.3 High Performance Computing libraries

Library packages that support environment modules follow a distinctive naming scheme. All packages have the compiler suite and, if built with MPI support, the MPI flavor included in their name: *-[MPI_FLAVOR-]COMPILER-hpc*. To allow the parallel installation of multiple versions of a library, the package name contains the version number (with dots . replaced by underscores _). master- packages are supplied to ensure that the latest version of a package is installed. When these master packages are updated, the latest version of the respective packages is installed, while leaving previous versions installed. Library packages are split between runtime and compile-time packages. The compile-time packages typically supply include files and .so files for shared libraries. Compile-time package names end with -devel. For some libraries, static (.a) libraries are supplied as well. Package names for these end with -devel-static.

As an example, these are the package names of the ScaLAPACK library version 2.0.2 built with GCC for Open MPI v2:

  • library package: libscalapack2_2_1_0-gnu-openmpi2-hpc

  • library master package: libscalapack2-gnu-openmpi2-hpc

  • development package: libscalapack2_2_1_0-gnu-openmpi2-hpc-devel

  • development master package: libscalapack2-gnu-openmpi2-hpc-devel

  • static library package: libscalapack2_2_1_0-gnu-openmpi2-hpc-devel-static

The digit 2 appended to the library name denotes the .so version of the library.

To install a library package, run zypper in LIBRARY-MASTER-PACKAGE. To install a development file, run zypper in LIBRARY-DEVEL-MASTER-PACKAGE.

The GNU compiler collection version 7 as provided with SLE HPC and the MPI flavors Open MPI v.3, Open MPI v.4, MPICH, and MVAPICH2 are currently supported.

The Development Tools Module might provide later versions of the GNU compiler suite. To view available compilers, run the following command:

# zypper search '*-compilers-hpc'

7.3.1 boost — Boost C++ Libraries

Boost is a set of portable C++ libraries that provide a reference implementation of "existing practices". See the full release notes for Boost 1.71 at https://www.boost.org/users/history/version_1_71_0.html.

To load the highest available serial version of this module, run the following command:

# module load TOOLCHAIN boost

To use MPI-specific boost libraries, run the following command:

# module load TOOLCHAIN MPI_FLAVOR boost

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • boost-gnu-hpc

  • boost-gnu-hpc-devel

Most Boost libraries do not depend on MPI flavors. However, Boost contains a set of libraries to abstract interaction with MPI. These libraries depend on the MPI flavor used.

List of master packages:

  • boost-gnu-MPI_FLAVOR-hpc

  • boost-gnu-MPI_FLAVOR-hpc-devel

  • boost-gnu-MPI_FLAVOR-hpc-python3

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

7.3.2 FFTW HPC library — discrete Fourier transforms

FFTW is a C subroutine library for computing the Discrete Fourier Transform (DFT) in one or more dimensions, of both real and complex data, and of arbitrary input size.

This library is available as both a serial and an MPI-enabled variant. This module requires a compiler toolchain module loaded. To select an MPI variant, the respective MPI module must be loaded beforehand. To load this module, run the following command:

# module load TOOLCHAIN fftw3

To load the MPI-enabled variant, run the following command:

# module load TOOLCHAIN MPI_FLAVOR fftw3

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • libfftw3-gnu-hpc

  • fftw3-gnu-hpc-devel

  • libfftw3-gnu-MPI_FLAVOR-hpc

  • fftw3-gnu-MPI_FLAVOR-hpc-devel

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

7.3.3 NumPy Python library

NumPy is a general-purpose array-processing package designed to efficiently manipulate large multi-dimensional arrays of arbitrary records without sacrificing too much speed for small multi-dimensional arrays.

NumPy is built on the Numeric code base and adds features introduced by the discontinued NumArray project, as well as an extended C API, and the ability to create arrays of arbitrary type, which also makes NumPy suitable for interfacing with general-purpose database applications.

There are also basic facilities for discrete Fourier transform, basic linear algebra, and random number generation.

This package is available both for Python 2 and 3. The specific compiler toolchain module must be loaded for this library. The correct library module for the Python version used needs to be specified when loading this module. To load this module, run the following command:

# module load TOOLCHAIN pythonVERSION-numpy

For information about the toolchain to load see: Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • pythonVERSION-numpy-gnu-hpc

  • pythonVERSION-numpy-gnu-hpc-devel

7.3.4 SciPy Python Library

SciPy is a collection of mathematical algorithms and convenience functions built on the NumPy extension of Python. It provides high-level commands and classes for manipulating and visualizing data. With SciPy, an interactive Python session becomes a data-processing and system-prototyping environment.

This package is available both for Python 2 (up to version 1.2.0 only) and 3. The specific compiler toolchain modules must be loaded for this library. The correct library module for the Python version used must be specified when loading this module. To load this module, run the following command:

# module load TOOLCHAIN pythonVERSION-scipy

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • pythonPYTHON_VERSION-scipy-gnu-hpc

  • pythonPYTHON_VERSION-scipy-gnu-hpc-devel

7.3.5 HYPRE — scalable linear solvers and multigrid methods

HYPRE is a library of linear solvers that are designed to solve large and detailed simulations faster than traditional methods at large scales.

The library offers a comprehensive suite of scalable solvers for large-scale scientific simulation, featuring parallel multigrid methods for both structured and unstructured grid problems. HYPRE is highly portable and supports several languages. It is developed at Lawrence Livermore National Laboratory.

For this library, a compiler toolchain and an MPI flavor needs to be loaded beforehand. To load this module, run the following command:

# module load TOOLCHAIN MPI_FLAVOR hypre

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • hypre-gnu-MPI_FLAVOR-hpc-devel

  • libHYPRE-gnu-MPI_FLAVOR-hpc

7.3.6 METIS — serial graph partitioning and fill-reducing matrix ordering library

METIS is a set of serial programs for partitioning graphs, partitioning finite element meshes, and producing fill-reducing orderings for sparse matrices. The algorithms implemented in METIS are based on the multilevel recursive-bisection, multilevel k-way, and multi-constraint partitioning schemes.

For this library, a compiler toolchain must be loaded beforehand. To load METIS, run the following command:

# module load TOOLCHAIN metis

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • metis-gnu-hpc

  • metis-gnu-hpc-devel

  • metis-gnu-hpc-doc

  • metis-gnu-hpc-examples

  • libmetis-gnu-hpc

7.3.7 GSL — GNU Scientific Library

The GNU Scientific Library (GSL) is a numerical library for C and C++ programmers.

The library provides a wide range of mathematical routines such as random number generators, special functions, and least-squares fitting. There are over 1000 functions in total with an extensive test suite.

It is free software under the GNU General Public License.

For this library, a compiler toolchain must be loaded beforehand. To load GSL, run the following command:

# module load TOOLCHAIN gsl

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • gsl-gnu-hpc

  • gsl-gnu-hpc-devel

  • gsl-gnu-hpc-doc

  • libgsl-gnu-hpc

  • libgslcblas-gnu-hpc

7.3.8 OCR — Open Community Runtime (OCR) for shared memory

The Open Community Runtime project is an application-building framework that explores methods for high-core-count programming with focus on HPC applications.

This first reference implementation is a functionally-complete implementation of the OCR 1.0.0 specification, with extensive tools and demonstration examples, running on both single nodes and clusters.

This library is available both without and with MPI support. For this library, a compiler toolchain, and if applicable an MPI flavor, must be loaded beforehand. To load ocr, run the following command:

# module load TOOLCHAIN ocr

To load ocr with MPI support, run the following command:

# module load TOOLCHAIN MPI_FLAVOR ocr

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • ocr-gnu-hpc

  • ocr-gnu-hpc-devel

  • ocr-gnu-hpc-doc

  • ocr-gnu-hpc-examples

  • ocr-gnu-MPI_FLAVOR-hpc

  • ocr-gnu-MPI_FLAVOR-hpc-devel

  • ocr-gnu-MPI_FLAVOR-hpc-doc

  • ocr-gnu-MPI_FLAVOR-hpc-examples

7.3.9 memkind — heap manager for heterogeneous memory platforms and mixed memory policies

The memkind library is a user-extensible heap manager built on top of jemalloc. It enables control over memory characteristics and a partitioning of the heap between kinds of memory. The kinds of memory are defined by operating system memory policies that have been applied to virtual address ranges. Memory characteristics supported by memkind without user extension include control of NUMA and page size features.

For more information, see:

Note

This tool is only available for AMD64/Intel 64.

7.3.10 MUMPS — MUltifrontal Massively Parallel sparse direct Solver

MUMPS (not to be confused with the database programming language with the same acronym) solves a sparse system of linear equations (A x = b) using Gaussian elimination.

This library requires a compiler toolchain and an MPI flavor to be loaded beforehand. To load this module, run the following command:

# module load TOOLCHAIN MPI_FLAVOR mumps

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • libmumps-gnu-MPI_FLAVOR-hpc

  • mumps-gnu-MPI_FLAVOR-hpc-devel

  • mumps-gnu-MPI_FLAVOR-hpc-doc

  • mumps-gnu-MPI_FLAVOR-hpc-examples

7.3.11 Support for PMIx in Slurm and MPI libraries

PMIx abstracts the internals of MPI implementations for workload managers and unifies the way MPI jobs are started by the workload manager. With PMIx, there is no need to use the individual MPI launchers on Slurm, because srun will take care of this. In addition, the workload manager can determine the topology of the cluster, so you do not need to specify topologies manually.

7.3.12 OpenBLAS library — optimized BLAS library

OpenBLAS is an optimized BLAS (Basic Linear Algebra Subprograms) library based on GotoBLAS2 1.3, BSD version. It provides the BLAS API. It is shipped as a package enabled for environment modules, so it requires using Lmod to select a version. There are two variants of this library: an OpenMP-enabled variant, and a pthreads variant.

OpenMP-Enabled Variant

The OpenMP variant covers the following use cases:

  • Programs using OpenMP. This requires the OpenMP-enabled library version to function correctly.

  • Programs using pthreads. This requires an OpenBLAS library without pthread support. This can be achieved with the OpenMP-version. We recommend limiting the number of threads that are used to 1 by setting the environment variable OMP_NUM_THREADS=1.

  • Programs without pthreads and without OpenMP. Such programs can still take advantage of the OpenMP optimization in the library by linking against the OpenMP variant of the library.

When linking statically, ensure that libgomp.a is included by adding the linker flag -lgomp.

pthreads Variant

The pthreads variant of the OpenBLAS library can improve the performance of single-threaded programs. The number of threads used can be controlled with the environment variable OPENBLAS_NUM_THREADS.

Installation and Usage

This module requires loading a compiler toolchain beforehand. To select the latest version of this module provided, run the following command:

  • Standard version:

    # module load TOOLCHAIN openblas
  • OpenMP/pthreads version:

    # module load TOOLCHAIN openblas-pthreads

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • libopenblas-gnu-hpc

  • libopenblas-gnu-hpc-devel

  • libopenblas-pthreads-gnu-hpc

  • libopenblas-pthreads-gnu-hpc-devel

7.3.13 PETSc HPC library — solver for partial differential equations

PETSc is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations.

This module requires loading a compiler toolchain and an MPI library flavor beforehand. To load this module, run the following command:

# module load TOOLCHAIN MPI_FLAVOR petsc

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • libpetsc-gnu-MPI_FLAVOR-hpc

  • petsc-gnu-MPI_FLAVOR-hpc-devel

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

7.3.14 ScaLAPACK HPC library — LAPACK routines

The library ScaLAPACK (short for Scalable LAPACK) includes a subset of LAPACK routines designed for distributed memory MIMD-parallel computers.

This library requires loading both a compiler toolchain and an MPI library flavor beforehand. To load this module, run the following command:

# module load TOOLCHAIN MPI_FLAVOR scalapack

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • libscalapack2-gnu-MPI_FLAVOR-hpc

  • libscalapack2-gnu-MPI_FLAVOR-hpc-devel

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

7.3.15 SCOTCH — static mapping and sparse matrix reordering algorithms

SCOTCH is a set of programs and libraries that implement the static mapping and sparse matrix reordering algorithms developed in the SCOTCH project.

SCOTCH applies graph theory to scientific computing problems such as graph and mesh partitioning, static mapping, and sparse matrix ordering, in application domains ranging from structural mechanics to operating systems or bio-chemistry.

For this library a compiler toolchain and an MPI flavor must be loaded beforehand. To load this module, run the following command:

# module load TOOLCHAIN MPI_FLAVOR scotch

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • libptscotch-gnu-MPI_FLAVOR-hpc

  • ptscotch-gnu-MPI_FLAVOR-hpc

  • ptscotch-gnu-MPI_FLAVOR-hpc-devel

7.3.16 SuperLU — supernodal LU decomposition of sparse matrices

SuperLU is a general-purpose library for the direct solution of large, sparse, nonsymmetric systems of linear equations. The library is written in C and can be called from C and Fortran programs.

This library requires a compiler toolchain and an MPI flavor to be loaded beforehand. To load this module, run the following command:

# module load TOOLCHAIN superlu

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • libsuperlu-gnu-hpc

  • superlu-gnu-hpc-devel

  • superlu-gnu-hpc-doc

  • superlu-gnu-hpc-examples

7.3.17 Trilinos — object-oriented software framework

The Trilinos Project is an effort to develop algorithms and enabling technologies within an object-oriented software framework for the solution of large-scale, complex multi-physics engineering and scientific problems. A unique design feature of Trilinos is its focus on packages.

This library needs a compiler toolchain and MPI flavor to be loaded beforehand. To load this module, run the following command:

# module load TOOLCHAIN MPI_FLAVOR trilinos

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • libtrilinos-gnu-MPI_FLAVOR-hpc

  • trilinos-gnu-MPI_FLAVOR-hpc-devel

7.4 File format libraries

7.4.1 Adaptable IO System (ADIOS)

The Adaptable IO System (ADIOS) provides a flexible way for scientists to describe the data in their code that might need to be written, read, or processed outside of the running simulation. For more information, see https://www.olcf.ornl.gov/center-projects/adios/.

To load this module, run the following command:

# module load TOOLCHAIN MPI_FLAVOR adios

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • adios-gnu-MPI_FLAVOR-hpc

  • adios-gnu-MPI_FLAVOR-hpc-devel

Replace MPI_FLAVOR with one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

7.4.2 HDF5 HPC library — model, library, and file format for storing and managing data

HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of data types, and is designed for flexible and efficient I/O and for high-volume and complex data. HDF5 is portable and extensible, allowing applications to evolve in their use of HDF5.

There are serial and MPI variants of this library available. All flavors require loading a compiler toolchain module beforehand. The MPI variants also require loading the correct MPI flavor module.

To load the highest available serial version of this module, run the following command:

# module load TOOLCHAIN hdf5

When an MPI flavor is loaded, you can load the MPI version of this module by running the following command:

# module load TOOLCHAIN MPI_FLAVOR phpdf5

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • hdf5-examples

  • hdf5-gnu-hpc-devel

  • libhdf5-gnu-hpc

  • libhdf5_cpp-gnu-hpc

  • libhdf5_fortran-gnu-hpc

  • libhdf5_hl_cpp-gnu-hpc

  • libhdf5_hl_fortran-gnu-hpc

  • hdf5-gnu-MPI_FLAVOR-hpc-devel

  • libhdf5-gnu-MPI_FLAVOR-hpc

  • libhdf5_fortran-gnu-MPI_FLAVOR-hpc

  • libhdf5_hl_fortran-MPI_FLAVOR-hpc

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

For general information about Lmod and modules, see Section 7.1, “Lmod — Lua-based environment modules”.

7.4.3 NetCDF HPC library — implementation of self-describing data formats

The NetCDF software libraries for C, C++, Fortran, and Perl are sets of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

netcdf Packages

The packages with names starting with netcdf provide C bindings for the NetCDF API. These are available with and without MPI support.

There are serial and MPI variants of this library available. All flavors require loading a compiler toolchain module beforehand. The MPI variants also require loading the correct MPI flavor module.

The MPI variant is available after the MPI module is loaded. Both variants require loading a compiler toolchain module beforehand. To load the highest version of the non-MPI netcdf module, run the following command:

# module load TOOLCHAIN MPI_FLAVOR netcdf

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • netcdf-gnu-hpc

  • netcdf-gnu-hpc-devel

  • netcdf-gnu-hpc

  • netcdf-gnu-hpc-devel

  • netcdf-gnu-MPI_FLAVOR-hpc

  • netcdf-gnu-MPI_FLAVOR-hpc-devel

  • netcdf-gnu-MPI_FLAVOR-hpc

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

netcdf-cxx Packages

netcdf-cxx4 provides a C++ binding for the NetCDF API.

This module requires loading a compiler toolchain module beforehand. To load this module, run the following command:

# module load TOOLCHAIN netcdf-cxx4

For information about the toolchain to load, see: Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • libnetcdf-cxx4-gnu-hpc

  • libnetcdf-cxx4-gnu-hpc-devel

  • netcdf-cxx4-gnu-hpc-tools

netcdf-fortran Packages

The netcdf-fortran packages provide Fortran bindings for the NetCDF API, with and without MPI support.

There are serial and MPI variants of this library available. All flavors require loading a compiler toolchain module beforehand. The MPI variants also require loading the correct MPI flavor module.

The MPI variant is available after the MPI module is loaded. Both variants require loading a compiler toolchain module beforehand. To load the highest version of the non-MPI netcdf-fortran module, run the following command:

# module load TOOLCHAIN netcdf-fortran

To load the MPI variant, run the following command:

# module load TOOLCHAIN MPI_FLAVOR netcdf-fortran

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”, For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • libnetcdf-fortran-gnu-hpc

  • libnetcdf-fortran-gnu-hpc-devel

  • libnetcdf-fortran-gnu-hpc

  • libnetcdf-fortran-gnu-MPI_FLAVOR-hpc

  • libnetcdf-fortran-gnu-MPI_FLAVOR-hpc-devel

  • libnetcdf-fortran-gnu-MPI_FLAVOR-hpc

7.4.4 HPC flavor of pnetcdf

NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

Parallel netCDF (PnetCDF) is a library providing high-performance I/O while still maintaining file-format compatibility with NetCDF by Unidata.

The package is available for the MPI Flavors Open MPI 2 and 3, MVAPICH2, and MPICH.

To load the highest available serial version of this module, run the following command:

# module load TOOLCHAIN MPI_FLAVOR pnetcdf

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of MPI master packages:

  • libpnetcdf-gnu-MPI_FLAVOR-hpc

  • pnetcdf-gnu-MPI_FLAVOR-hpc

  • pnetcdf-gnu-MPI_FLAVOR-hpc-devel

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

7.5 MPI libraries

Three different implementation of the Message Passing Interface (MPI) standard are provided standard with the HPC module:

  • Open MPI (version 3 and version 4)

  • MVAPICH2

  • MPICH

These packages have been built with full environment module support (LMOD).

The following packages are available:

  • For Open MPI:

    • user programs: openmpi3-gnu-hpc and openmpi4-gnu-hpc

    • shared libraries: libopenmpi3-gnu-hpc and libopenmpi4-gnu-hpc

    • development libraries, headers and tools required for building: openmpi3-gnu-hpc-devel and openmpi4-gnu-hpc-devel

    • documentation: openmpi3-gnu-hpc-docs and openmpi4-gnu-hpc-docs.

  • For MVAPICH2

    • user programs and libraries: mvapich2-gnu-hpc

    • development libraries, headers and tools for building: mvapich2-gnu-hpc-devel

    • documentation: mvapich2-gnu-hpc-doc

    For MPICH:

    • user programs and libraries: mpich-gnu-hpc

    • development libraries, headers and tools for building: mpich-gnu-hpc-devel

The different MPI implementations and versions are independent of each other, and can be installed in parallel.

Use environment modules to pick the version to use:

  • For Open MPI v.3:

    # module load TOOLCHAIN openmpi/3
  • For Open MPI v.4:

    # module load TOOLCHAIN openmpi/4
  • For MVAPICH2:

    # module load TOOLCHAIN mvapich2
  • For MPICH:

    # module load TOOLCHAIN mpich

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

7.6 Profiling and benchmarking libraries and tools

SUSE Linux Enterprise High Performance Computing provides tools for profiling MPI applications and benchmarking MPI performance.

7.6.1 IMB — Intel* MPI benchmarks

The Intel* MPI Benchmarks package provides a set of elementary benchmarks that conform to the MPI-1, MPI-2, and MPI-3 standards. You can run all of the supported benchmarks, or a subset specified in the command line, using a single executable file. Use command line parameters to specify various settings, such as time measurement, message lengths, and selection of communicators. For details, see the Intel* MPI Benchmarks User's Guide: https://software.intel.com/en-us/imb-user-guide.

For the IMB binaries to be found, a compiler toolchain and an MPI flavor must be loaded beforehand. To load this module, run the following command:

# module load TOOLCHAIN MPI_FLAVOR imb

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

  • imb-gnu-MPI_FLAVOR-hpc

7.6.2 PAPI HPC library — consistent interface for hardware performance counters

PAPI provides a tool with a consistent interface and methodology for the performance counter hardware found in most major microprocessors.

This package works with all compiler toolchains and does not require a compiler toolchain to be selected. Load the latest version provided by running the following command:

# module load TOOLCHAIN papi

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • papi-hpc

  • papi-hpc-devel

For general information about Lmod and modules, see Section 7.1, “Lmod — Lua-based environment modules”.

7.6.3 mpiP — lightweight MPI profiling library

mpiP is a lightweight profiling library for MPI applications. Because it only collects statistical information about MPI functions, mpiP generates considerably less overhead and much less data than tracing tools. All the information captured by mpiP is task-local. It only uses communication during report generation, typically at the end of the experiment, to merge results from all of the tasks into one output file.

For this library a compiler toolchain and MPI flavor must be loaded beforehand. To load this module, run the following command:

# module load TOOLCHAIN MPI_FLAVOR mpip

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • mpiP-gnu-MPI_FLAVOR-hpc

  • mpiP-gnu-MPI_FLAVOR-hpc-devel

  • mpiP-gnu-MPI_FLAVOR-hpc-doc

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

Print this page