The Genders database in SUSE Linux Enterprise High Performance Computing is a plain-text file called /etc/genders. It contains a list of node names with their attributes. Each line of the database can have one of the following formats.

nodename                attr[=value],attr[=value],...
nodename1,nodename2,... attr[=value],attr[=value],...
nodenames[A-B]          attr[=value],attr[=value],...

Node names are listed without their domain, and are followed by any number of spaces or tabs, then the comma-separated list of attributes. Every attribute can optionally have a value. The substitution string %n can be used in an attribute value to represent the node name. Node names can be listed on multiple lines, so a node's attributes can be specified on multiple lines. However, no single node can have duplicate attributes.

The attribute list must not contain spaces, and there is no provision for continuation lines. Commas and equals characters (=) are special, and cannot appear in attribute names or values. Comments are prefixed with the hash character (#) and can appear anywhere in the file.

Ranges for node names can be specified in the form prefix[a-c,n-p,...] as an alternative to explicit lists of node names. For example, node[01-03,06] would specify node01, node02, node03, and node06.

The command line utility nodeattr can be used to query data in the genders file. When the genders file is replicated on all nodes, a query can be done without network access. The genders file can be called as follows:

> nodeattr [-q | -n | -s] [-r] attr[=val]

-q is the default option and prints a list of nodes with attr[=val].

The -c or -s options give a comma-separated or space-separated list of nodes with attr[=val].

If none of the formatting options are specified, nodeattr returns a zero value if the local node has the specified attribute, and non-zero otherwise. The -v option causes any value associated with the attribute to go to stdout. If a node name is specified before the attribute, the specified node is queried instead of the local node.

To print all attributes for a particular node, run the following command:

> nodeattr -l [node]

If no node parameter is given, all attributes of the local node are printed.

To perform a syntax check of the genders database, run the following command:

> nodeattr [-f genders] -k

To specify an alternative database location, use the option -f.

MUNGE uses UID/GID values to uniquely identify and authenticate users, so you must ensure that users who will authenticate across a network have matching UIDs and GIDs across all nodes.

MUNGE credentials have a limited time-to-live, so you must ensure that the time is synchronized across the entire cluster.

MUNGE is installed with the command zypper in munge. This also installs further required packages. A separate munge-devel package is available to build applications that require MUNGE authentication.

When installing the munge package, a new key is generated on every system. However, the entire cluster needs to use the same MUNGE key. Therefore, you must securely copy the MUNGE key from one system to all the other nodes in the cluster. You can accomplish this by using pdsh with SSH. Ensure that the key is only readable by the munge user (permissions mask 0400).

munged must be running on all nodes that use MUNGE authentication. If MUNGE is used for authentication across the network, it needs to run on each side of the communications link.

To start the service and ensure it is started after every reboot, run the following command on each node:

> sudo systemctl enable --now munge.service

You can also use pdsh to run this command on multiple nodes at once.

In a minimal installation, Slurm only stores pending and running jobs. To store finished and failed job data, the storage plugin must be installed and enabled. You can also enable completely fair scheduling, which replaces FIFO (first in, first out) scheduling with algorithms that calculate the job priority in a queue in dependence of the job which a user has run in the history.

The Slurm database has two components: the slurmdbd daemon itself, and an SQL database. MariaDB is recommended. The database can be installed on the same node that runs slurmdbd, or on a separate node. For a minimal setup, all these services run on the management server.

Before you begin, make sure Slurm is installed as described in Section 5.1.1, “Minimal installation”.

Procedure 5.4: Install slurmdbd #

  

Note: MariaDB

If you want to use an external SQL database (or you already have a database installed on the management server), you can skip Step 1 and Step 2.

1. Install the MariaDB SQL database:

   management# zypper install mariadb

2. Start and enable MariaDB:

   management# systemctl enable --now mariadb

3. Secure the database:

   management# mysql_secure_installation

4. Connect to the SQL database:

   management# mysql -u root -p

5. Create the Slurm database user and grant it permissions for the Slurm database, which will be created later:

   mysql> create user 'slurm'@'localhost' identified by 'PASSWORD';
   mysql> grant all on slurm_acct_db.* TO 'slurm'@'localhost';

   You can choose to use a different user name or database name. In this case, you must also change the corresponding values in the /etc/slurm/slurmdbd.conf file later.

6. Exit the database:

   mysql> exit

7. Install the slurmdbd package:

   management# zypper in slurm-slurmdbd

8. Edit the /etc/slurm/slurmdbd.conf file so that the daemon can access the database. Change the following line to the password that you used in Step 5:

   StoragePass=password

   If the database is on a different node, or if you chose a different user name or database name, you must also modify the following lines:

   StorageUser=slurm
   StorageLoc=slurm_acct_db
   DbdAddr=localhost
   DbdHost=localhost

9. Start and enable slurmdbd:

   management# systemctl enable --now slurmdbd

   The first start of slurmdbd might take some time.

10. To enable accounting, edit the /etc/slurm/slurm.conf file to add the connection between slurmctld and the slurmdbd daemon. Ensure that the following lines appear as shown:

    JobAcctGatherType=jobacct_gather/linux
    JobAcctGatherFrequency=30
    AccountingStorageType=accounting_storage/slurmdbd
    AccountingStorageHost=localhost

    

    This example assumes that slurmdbd is running on the same node as slurmctld. If not, change localhost to the host name or IP address of the node where slurmdbd is running.

11. Make sure slurmdbd is running before you continue:

    management# systemctl status slurmdbd

    If you restart slurmctld before slurmdbd is running, slurmctld fails because it cannot connect to the database.

12. Restart slurmctld:

    management# systemctl restart slurmctld

    This creates the Slurm database and adds the cluster to the database (using the ClusterName from /etc/slurm/slurm.conf).

13. (Optional) By default, Slurm does not take any group membership into account, and the system groups cannot be mapped to Slurm. However, you can mimic system groups with accounts. In Slurm, accounts are usually entities billed for cluster usage, while users identify individual cluster users. Multiple users can be associated with a single account.

    The following example creates an umbrella group bavaria for two subgroups called nuremberg and munich:

    management# sacctmgr add account bavaria \
    Description="umbrella group for subgroups" Organization=bavaria

    management# sacctmgr add account nuremberg,munich parent=bavaria \
    Description="subgroup" Organization=bavaria

    The following example adds a user called tux to the subgroup nuremberg:

    management# sacctmgr add user tux Account=nuremberg

The command scontrol is used to show and update the entities of Slurm, such as the state of the compute nodes or compute jobs. It can also be used to reboot or to propagate configuration changes to the compute nodes.

Useful options for this command are --details, which prints more verbose output, and --oneliner, which forces the output onto a single line, which is more useful in shell scripts.

For more information, see man scontrol.

The command sinfo retrieves information about the state of the compute nodes, and can be used for a fast overview of the cluster health. For more information, see man sinfo.

These commands are used for managing accounting. For more information, see man sacctmgr and man sacct.

These commands are used to schedule compute jobs, which means batch scripts for the sbatch command, interactive sessions for the salloc command, or binaries for the srun command. If the job cannot be scheduled immediately, only sbatch places it into the queue.

For more information, see man sbatch, man salloc, and man srun.

Interoperability is guaranteed between three consecutive versions of Slurm, with the following restrictions:

Or in short: version(slurmdbd) >= version(slurmctld) >= version(slurmd) >= version (Slurm user CLIs).

Slurm uses a segmented version number: the first two segments denote the major version, and the final segment denotes the patch level. Upgrade packages (that is, packages that were not initially supplied with the module or service pack) have their major version encoded in the package name (with periods . replaced by underscores _). For example, for version 23.02, this would be slurm_23_02-*.rpm. To find out the latest version of Slurm, you can check My Tools › Packages in the SUSE Customer Center, or run zypper search -v slurm on a node.

With each version, configuration options for slurmctld, slurmd, or slurmdbd might be deprecated. While deprecated, they remain valid for this version and the two consecutive versions, but they might be removed later. Therefore, it is advisable to update the configuration files after the upgrade and replace deprecated configuration options before the final restart of a service.

A new major version of Slurm introduces a new version of libslurm. Older versions of this library might not work with an upgraded Slurm. An upgrade is provided for all SUSE Linux Enterprise software that depends on libslurm . It is strongly recommended to rebuild local applications using libslurm, such as MPI libraries with Slurm support, as early as possible. This might require updating the user applications, as new arguments might be introduced to existing functions.

Warning: Upgrade slurmdbd databases before other Slurm components

If slurmdbd is used, always upgrade the slurmdbd database before starting the upgrade of any other Slurm component. The same database can be connected to multiple clusters and must be upgraded before all of them.

Upgrading other Slurm components before the database can lead to data loss.

When upgrading slurmdbd, the database is converted when the new version of slurmdbd starts for the first time. If the database is big, the conversion could take several tens of minutes. During this time, the database is inaccessible.

It is highly recommended to create a backup of the database in case an error occurs during or after the upgrade process. Without a backup, all accounting data collected in the database might be lost if an error occurs or the upgrade is rolled back. A database converted to a newer version cannot be converted back to an older version, and older versions of slurmdbd do not recognize the newer formats.

Note: Convert primary slurmdbd first

If you are using a backup slurmdbd, the conversion must be performed on the primary slurmdbd first. The backup slurmdbd only starts after the conversion is complete.

After the Slurm database is upgraded, the slurmctld and slurmd instances can be upgraded. It is recommended to update the management servers and compute nodes all at once. If this is not feasible, the compute nodes (slurmd) can be updated on a node-by-node basis. However, the management servers (slurmctld) must be updated first.

Install Prometheus and Grafana on a management server, or on a separate monitoring node.

You can now configure Prometheus to collect metrics from the cluster, and add dashboards to Grafana to visualize those metrics.

To monitor the status of the nodes and jobs in an HPC cluster, install the Prometheus Slurm exporter to collect workload data, then import a custom Slurm dashboard from the Grafana website to visualize the data. For more information about this dashboard, see https://grafana.com/grafana/dashboards/4323.

You must install the Slurm exporter on a node that has access to the Slurm command line interface. In the following procedure, the Slurm exporter will be installed on a management server.

The Slurm dashboard is now available from the Home screen in Grafana.

To monitor the performance and health of each compute node, install the Prometheus node exporter to collect performance data, then import a custom node dashboard from the Grafana website to visualize the data. For more information about this dashboard, see https://grafana.com/grafana/dashboards/405.

The node dashboard is now available from the Home screen in Grafana.

To use Ganglia, install ganglia-gmetad on the management server, then start the Ganglia meta-daemon: rcgmead start. To make sure the service is started after a reboot, run: systemctl enable gmetad. On each cluster node which you want to monitor, install ganglia-gmond, start the service rcgmond start and make sure it is enabled to start automatically after a reboot: systemctl enable gmond. To test whether the gmond daemon has connected to the meta-daemon, run gstat -a and check that each node to be monitored is present in the output.

When using the Btrfs file system, the monitoring data will be lost after a rollback of the service gmetad. To fix this issue, either install the package ganglia-gmetad-skip-bcheck or create the file /etc/ganglia/no_btrfs_check.

Install ganglia-web on the management server. Enable PHP in Apache2: a2enmod php7. Then start Apache2 on this machine: rcapache2 start and make sure it is started automatically after a reboot: systemctl enable apache2. The Ganglia Web interface is accessible from http://MANAGEMENT_SERVER/ganglia.

To install Lmod, run zypper in lua-lmod.

Before you can use Lmod, you must source an init file into the initialization file of your interactive shell. The following init files are available for various common shells:

/usr/share/lmod/lmod/init/bash
/usr/share/lmod/lmod/init/ksh
/usr/share/lmod/lmod/init/tcsh
/usr/share/lmod/lmod/init/zsh
/usr/share/lmod/lmod/init/sh

Pick the appropriate file for your shell, then add the following line into your shell's init file:

source /usr/share/lmod/lmod/init/INIT-FILE

The init script adds the command module.

To list all the available modules, run module spider. To show all modules which can be loaded with the currently loaded modules, run module avail. A module name consists of a name and a version string, separated by a / character. If more than one version is available for a certain module name, the default version is marked by a * character. If there is no default, the module with the highest version number is loaded. To reference a specific module version, you can use the full string NAME/VERSION.

module list shows all currently loaded modules. Refer to module help for some short help on the module command, and module help MODULE-NAME for help on the particular module. The module command is only available when you log in after installing lua-lmod.

To get information about a particular module, run module whatis MODULE-NAME. To load a module, run module load MODULE-NAME. This will ensure that your environment is modified (that is, the PATH and LD_LIBRARY_PATH and other environment variables are prepended) so that binaries and libraries provided by the respective modules are found. To run a program compiled against this library, the appropriate module load commands must be issued beforehand.

The module load MODULE command must be run in the shell from which the module is to be used. Some modules require a compiler toolchain or MPI flavor module to be loaded before they are available for loading.

If the respective development packages are installed, build-time environment variables like LIBRARY_PATH, CPATH, C_INCLUDE_PATH, and CPLUS_INCLUDE_PATH are set up to include the directories containing the appropriate header and library files. However, some compiler and linker commands might not honor these. In this case, use the appropriate options together with the environment variables -I PACKAGE_NAME_INC and -L PACKAGE_NAME_LIB to add the include and library paths to the command lines of the compiler and linker.

For more information on Lmod, see https://lmod.readthedocs.org.

This package requires lua-lmod to supply environment module support.

To install gnu-compilers-hpc, run the following command:

> sudo zypper in gnu-compilers-hpc

To make libraries built with the base compilers available, you must set up the environment appropriately and select the GNU toolchain. To do so, run the following command:

> module load gnu

To use the GNU compiler collection to build your own libraries and applications, gnu-compilers-hpc-devel must be installed. It ensures that all compiler components required for HPC (that is, C, C++, and Fortran compilers) are installed.

The environment variables CC, CXX, FC and F77 will be set correctly and the path will be adjusted so that the correct compiler version can be found.

The Development Tools Module might provide later versions of the GNU compiler suite. To determine the available compiler suites, run the following command:

> zypper search '*-compilers-hpc'

If you have more than one version of the compiler suite installed, Lmod picks the latest one by default. If you require an older version, or the base version, append the version number:

> module load gnu/7

For more information, see Section 7.1, “Lmod — Lua-based environment modules”.

Boost is a set of portable C++ libraries that provide a reference implementation of "existing practices". See the full release notes for Boost 1.71 at https://www.boost.org/users/history/version_1_71_0.html.

To load the highest available serial version of this module, run the following command:

> module load TOOLCHAIN boost

To use MPI-specific boost libraries, run the following command:

> module load TOOLCHAIN MPI_FLAVOR boost

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

Most Boost libraries do not depend on MPI flavors. However, Boost contains a set of libraries to abstract interaction with MPI. These libraries depend on the MPI flavor used.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

FFTW is a C subroutine library for computing the Discrete Fourier Transform (DFT) in one or more dimensions, of both real and complex data, and of arbitrary input size.

This library is available as both a serial and an MPI-enabled variant. This module requires a compiler toolchain module loaded. To select an MPI variant, the respective MPI module must be loaded beforehand. To load this module, run the following command:

> module load TOOLCHAIN fftw3

To load the MPI-enabled variant, run the following command:

> module load TOOLCHAIN MPI_FLAVOR fftw3

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

NumPy is a general-purpose array-processing package designed to efficiently manipulate large multi-dimensional arrays of arbitrary records without sacrificing too much speed for small multi-dimensional arrays.

NumPy is built on the Numeric code base and adds features introduced by the discontinued NumArray project, as well as an extended C API, and the ability to create arrays of arbitrary type, which also makes NumPy suitable for interfacing with general-purpose database applications.

There are also basic facilities for discrete Fourier transform, basic linear algebra, and random number generation.

This package is available both for Python 2 and 3. The specific compiler toolchain module must be loaded for this library. The correct library module for the Python version used needs to be specified when loading this module. To load this module, run the following command:

> module load TOOLCHAIN pythonVERSION-numpy

For information about the toolchain to load see: Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

SciPy is a collection of mathematical algorithms and convenience functions built on the NumPy extension of Python. It provides high-level commands and classes for manipulating and visualizing data. With SciPy, an interactive Python session becomes a data-processing and system-prototyping environment.

This package is available both for Python 2 (up to version 1.2.0 only) and 3. The specific compiler toolchain modules must be loaded for this library. The correct library module for the Python version used must be specified when loading this module. To load this module, run the following command:

> module load TOOLCHAIN pythonVERSION-scipy

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

HYPRE is a library of linear solvers that are designed to solve large and detailed simulations faster than traditional methods at large scales.

The library offers a comprehensive suite of scalable solvers for large-scale scientific simulation, featuring parallel multigrid methods for both structured and unstructured grid problems. HYPRE is highly portable and supports several languages. It is developed at Lawrence Livermore National Laboratory.

For this library, a compiler toolchain and an MPI flavor needs to be loaded beforehand. To load this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR hypre

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

METIS is a set of serial programs for partitioning graphs, partitioning finite element meshes, and producing fill-reducing orderings for sparse matrices. The algorithms implemented in METIS are based on the multilevel recursive-bisection, multilevel k-way, and multi-constraint partitioning schemes.

For this library, a compiler toolchain must be loaded beforehand. To load METIS, run the following command:

> module load TOOLCHAIN metis

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

The GNU Scientific Library (GSL) is a numerical library for C and C++ programmers.

The library provides a wide range of mathematical routines such as random number generators, special functions, and least-squares fitting. There are over 1000 functions in total with an extensive test suite.

It is free software under the GNU General Public License.

For this library, a compiler toolchain must be loaded beforehand. To load GSL, run the following command:

> module load TOOLCHAIN gsl

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

The Open Community Runtime project is an application-building framework that explores methods for high-core-count programming with focus on HPC applications.

This first reference implementation is a functionally-complete implementation of the OCR 1.0.0 specification, with extensive tools and demonstration examples, running on both single nodes and clusters.

This library is available both without and with MPI support. For this library, a compiler toolchain, and if applicable an MPI flavor, must be loaded beforehand. To load ocr, run the following command:

> module load TOOLCHAIN ocr

To load ocr with MPI support, run the following command:

> module load TOOLCHAIN MPI_FLAVOR ocr

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

The memkind library is a user-extensible heap manager built on top of jemalloc. It enables control over memory characteristics and a partitioning of the heap between kinds of memory. The kinds of memory are defined by operating system memory policies that have been applied to virtual address ranges. Memory characteristics supported by memkind without user extension include control of NUMA and page size features.

For more information, see:

This tool is only available for AMD64/Intel 64.

MUMPS (not to be confused with the database programming language with the same acronym) solves a sparse system of linear equations (A x = b) using Gaussian elimination.

This library requires a compiler toolchain and an MPI flavor to be loaded beforehand. To load this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR mumps

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

PMIx abstracts the internals of MPI implementations for workload managers and unifies the way MPI jobs are started by the workload manager. With PMIx, there is no need to use the individual MPI launchers on Slurm, because srun will take care of this. In addition, the workload manager can determine the topology of the cluster, so you do not need to specify topologies manually.

OpenBLAS is an optimized BLAS (Basic Linear Algebra Subprograms) library based on GotoBLAS2 1.3, BSD version. It provides the BLAS API. It is shipped as a package enabled for environment modules, so it requires using Lmod to select a version. There are two variants of this library: an OpenMP-enabled variant, and a pthreads variant.

The OpenMP variant covers the following use cases:

When linking statically, ensure that libgomp.a is included by adding the linker flag -lgomp.

The pthreads variant of the OpenBLAS library can improve the performance of single-threaded programs. The number of threads used can be controlled with the environment variable OPENBLAS_NUM_THREADS.

This module requires loading a compiler toolchain beforehand. To select the latest version of this module provided, run the following command:

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

PETSc is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations.

This module requires loading a compiler toolchain and an MPI library flavor beforehand. To load this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR petsc

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

The library ScaLAPACK (short for Scalable LAPACK) includes a subset of LAPACK routines designed for distributed memory MIMD-parallel computers.

This library requires loading both a compiler toolchain and an MPI library flavor beforehand. To load this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR scalapack

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

SCOTCH is a set of programs and libraries that implement the static mapping and sparse matrix reordering algorithms developed in the SCOTCH project.

SCOTCH applies graph theory to scientific computing problems such as graph and mesh partitioning, static mapping, and sparse matrix ordering, in application domains ranging from structural mechanics to operating systems or bio-chemistry.

For this library a compiler toolchain and an MPI flavor must be loaded beforehand. To load this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR scotch

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

SuperLU is a general-purpose library for the direct solution of large, sparse, nonsymmetric systems of linear equations. The library is written in C and can be called from C and Fortran programs.

This library requires a compiler toolchain and an MPI flavor to be loaded beforehand. To load this module, run the following command:

> module load TOOLCHAIN superlu

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

The Trilinos Project is an effort to develop algorithms and enabling technologies within an object-oriented software framework for the solution of large-scale, complex multi-physics engineering and scientific problems. A unique design feature of Trilinos is its focus on packages.

This library needs a compiler toolchain and MPI flavor to be loaded beforehand. To load this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR trilinos

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

The Adaptable IO System (ADIOS) provides a flexible way for scientists to describe the data in their code that might need to be written, read, or processed outside of the running simulation. For more information, see https://www.olcf.ornl.gov/center-projects/adios/.

To load this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR adios

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

Replace MPI_FLAVOR with one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of data types, and is designed for flexible and efficient I/O and for high-volume and complex data. HDF5 is portable and extensible, allowing applications to evolve in their use of HDF5.

There are serial and MPI variants of this library available. All flavors require loading a compiler toolchain module beforehand. The MPI variants also require loading the correct MPI flavor module.

To load the highest available serial version of this module, run the following command:

> module load TOOLCHAIN hdf5

When an MPI flavor is loaded, you can load the MPI version of this module by running the following command:

> module load TOOLCHAIN MPI_FLAVOR phdf5

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

For general information about Lmod and modules, see Section 7.1, “Lmod — Lua-based environment modules”.

The NetCDF software libraries for C, C++, Fortran, and Perl are sets of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

The packages with names starting with netcdf provide C bindings for the NetCDF API. These are available with and without MPI support.

There are serial and MPI variants of this library available. All flavors require loading a compiler toolchain module beforehand. The MPI variants also require loading the correct MPI flavor module.

The MPI variant is available after the MPI module is loaded. Both variants require loading a compiler toolchain module beforehand. To load the highest version of the non-MPI netcdf module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR netcdf

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

netcdf-cxx4 provides a C++ binding for the NetCDF API.

This module requires loading a compiler toolchain module beforehand. To load this module, run the following command:

> module load TOOLCHAIN netcdf-cxx4

For information about the toolchain to load, see: Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

The netcdf-fortran packages provide Fortran bindings for the NetCDF API, with and without MPI support.

There are serial and MPI variants of this library available. All flavors require loading a compiler toolchain module beforehand. The MPI variants also require loading the correct MPI flavor module.

The MPI variant is available after the MPI module is loaded. Both variants require loading a compiler toolchain module beforehand. To load the highest version of the non-MPI netcdf-fortran module, run the following command:

> module load TOOLCHAIN netcdf-fortran

To load the MPI variant, run the following command:

> module load TOOLCHAIN MPI_FLAVOR netcdf-fortran

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”, For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

NetCDF is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data.

Parallel netCDF (PnetCDF) is a library providing high-performance I/O while still maintaining file-format compatibility with NetCDF by Unidata.

The package is available for the MPI Flavors Open MPI 2 and 3, MVAPICH2, and MPICH.

To load the highest available serial version of this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR pnetcdf

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of MPI master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

The Intel* MPI Benchmarks package provides a set of elementary benchmarks that conform to the MPI-1, MPI-2, and MPI-3 standards. You can run all of the supported benchmarks, or a subset specified in the command line, using a single executable file. Use command line parameters to specify various settings, such as time measurement, message lengths, and selection of communicators. For details, see the Intel* MPI Benchmarks User's Guide: https://software.intel.com/en-us/imb-user-guide.

For the IMB binaries to be found, a compiler toolchain and an MPI flavor must be loaded beforehand. To load this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR imb

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

PAPI provides a tool with a consistent interface and methodology for the performance counter hardware found in most major microprocessors.

This package works with all compiler toolchains and does not require a compiler toolchain to be selected. Load the latest version provided by running the following command:

> module load TOOLCHAIN papi

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

For general information about Lmod and modules, see Section 7.1, “Lmod — Lua-based environment modules”.

mpiP is a lightweight profiling library for MPI applications. Because it only collects statistical information about MPI functions, mpiP generates considerably less overhead and much less data than tracing tools. All the information captured by mpiP is task-local. It only uses communication during report generation, typically at the end of the experiment, to merge results from all of the tasks into one output file.

For this library a compiler toolchain and MPI flavor must be loaded beforehand. To load this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR mpip

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.