Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
documentation.suse.com / Administration Guide / HPC user libraries
Applies to SUSE Linux Enterprise High Performance Computing 15 SP4

7 HPC user libraries

Many HPC clusters need to accommodate multiple compute applications, each of which has its own very specific library dependencies. Multiple instances of the same libraries might exist, differing in version, build configuration, compiler, and MPI implementation. To manage these dependencies, you can use an environment module system. Most HPC libraries provided with SUSE Linux Enterprise High Performance Computing are built with support for environment modules. This chapter describes the environment module system Lmod, and a set of HPC compute libraries shipped with SLE HPC.

7.1 Lmod — Lua-based environment modules

Lmod is an advanced environment module system that allows the installation of multiple versions of a program or shared library, and helps configure the system environment for the use of a specific version. It supports hierarchical library dependencies and makes sure that the correct versions of dependent libraries are selected. Environment module-enabled library packages supplied with the HPC module support parallel installation of different versions and flavors of the same library or binary and are supplied with appropriate lmod module files.

7.1.1 Installation and basic usage

To install Lmod, run zypper in lua-lmod.

Before you can use Lmod, you must source an init file into the initialization file of your interactive shell. The following init files are available for various common shells:

/usr/share/lmod/lmod/init/bash
/usr/share/lmod/lmod/init/ksh
/usr/share/lmod/lmod/init/tcsh
/usr/share/lmod/lmod/init/zsh
/usr/share/lmod/lmod/init/sh

Pick the appropriate file for your shell, then add the following line into your shell's init file:

source /usr/share/lmod/lmod/init/INIT-FILE

The init script adds the command module.

7.1.2 Listing available modules

To list all the available modules, run module spider. To show all modules which can be loaded with the currently loaded modules, run module avail. A module name consists of a name and a version string, separated by a / character. If more than one version is available for a certain module name, the default version is marked by a * character. If there is no default, the module with the highest version number is loaded. To reference a specific module version, you can use the full string NAME/VERSION.

7.1.3 Listing loaded modules

module list shows all currently loaded modules. Refer to module help for some short help on the module command, and module help MODULE-NAME for help on the particular module. The module command is only available when you log in after installing lua-lmod.

7.1.4 Gathering information about a module

To get information about a particular module, run module whatis MODULE-NAME. To load a module, run module load MODULE-NAME. This will ensure that your environment is modified (that is, the PATH and LD_LIBRARY_PATH and other environment variables are prepended) so that binaries and libraries provided by the respective modules are found. To run a program compiled against this library, the appropriate module load commands must be issued beforehand.

7.1.5 Loading modules

The module load MODULE command must be run in the shell from which the module is to be used. Some modules require a compiler toolchain or MPI flavor module to be loaded before they are available for loading.

7.1.6 Environment variables

If the respective development packages are installed, build-time environment variables like LIBRARY_PATH, CPATH, C_INCLUDE_PATH, and CPLUS_INCLUDE_PATH are set up to include the directories containing the appropriate header and library files. However, some compiler and linker commands might not honor these. In this case, use the appropriate options together with the environment variables -I PACKAGE_NAME_INC and -L PACKAGE_NAME_LIB to add the include and library paths to the command lines of the compiler and linker.

7.1.7 For more information

For more information on Lmod, see https://lmod.readthedocs.org.

7.2 GNU Compiler Toolchain Collection for HPC

In SUSE Linux Enterprise High Performance Computing, the GNU compiler collection version 7 is provided as the base compiler toolchain. The gnu-compilers-hpc package provides the environment module for the base version of the GNU compiler suite. This package must be installed when using any of the HPC libraries enabled for environment modules.

7.2.1 Environment module

This package requires lua-lmod to supply environment module support.

To install gnu-compilers-hpc, run the following command:

> sudo zypper in gnu-compilers-hpc

To make libraries built with the base compilers available, you must set up the environment appropriately and select the GNU toolchain. To do so, run the following command:

> module load gnu

7.2.2 Building High Performance Computing software with GNU Compiler Suite

To use the GNU compiler collection to build your own libraries and applications, gnu-compilers-hpc-devel must be installed. It ensures that all compiler components required for HPC (that is, C, C++, and Fortran compilers) are installed.

The environment variables CC, CXX, FC and F77 will be set correctly and the path will be adjusted so that the correct compiler version can be found.

7.2.3 Later versions

The Development Tools Module might provide later versions of the GNU compiler suite. To determine the available compiler suites, run the following command:

> zypper search '*-compilers-hpc'

If you have more than one version of the compiler suite installed, Lmod picks the latest one by default. If you require an older version, or the base version, append the version number:

> module load gnu/7

For more information, see Section 7.1, “Lmod — Lua-based environment modules”.

7.3 High Performance Computing libraries

Library packages that support environment modules follow a distinctive naming scheme. All packages have the compiler suite and, if built with MPI support, the MPI flavor included in their name: *-[MPI_FLAVOR-]COMPILER-hpc*. To allow the parallel installation of multiple versions of a library, the package name contains the version number (with dots . replaced by underscores _). master- packages are supplied to ensure that the latest version of a package is installed. When these master packages are updated, the latest version of the respective packages is installed, while leaving previous versions installed. Library packages are split between runtime and compile-time packages. The compile-time packages typically supply include files and .so files for shared libraries. Compile-time package names end with -devel. For some libraries, static (.a) libraries are supplied as well. Package names for these end with -devel-static.

As an example, these are the package names of the ADIOS library version 1.13.1, built with GCC for Open MPI v4:

  • library master package: adios-gnu-openmpi4-hpc

  • development master package: adios-gnu-openmpi4-hpc-devel

  • library package: adios_1_13_1-gnu-openmpi4-hpc

  • development package: adios_1_13_1-gnu-openmpi4-hpc-devel

  • static library package: adios_1_13_1-gnu-openmpi4-hpc-devel-static

To install a library package, run zypper in LIBRARY-MASTER-PACKAGE. To install a development file, run zypper in LIBRARY-DEVEL-MASTER-PACKAGE.

The GNU compiler collection version 7 as provided with SLE HPC and the MPI flavors Open MPI v.3, Open MPI v.4, MPICH, and MVAPICH2 are currently supported.

The Development Tools Module might provide later versions of the GNU compiler suite. To view available compilers, run the following command:

> zypper search '*-compilers-hpc'

7.3.1 NumPy Python library

NumPy is a general-purpose array-processing package designed to efficiently manipulate large multi-dimensional arrays of arbitrary records without sacrificing too much speed for small multi-dimensional arrays.

NumPy is built on the Numeric code base and adds features introduced by the discontinued NumArray project, as well as an extended C API, and the ability to create arrays of arbitrary type, which also makes NumPy suitable for interfacing with general-purpose database applications.

There are also basic facilities for discrete Fourier transform, basic linear algebra, and random number generation.

This package is available both for Python 2 and 3. The specific compiler toolchain module must be loaded for this library. The correct library module for the Python version used needs to be specified when loading this module. To load this module, run the following command:

> module load TOOLCHAIN pythonVERSION-numpy

For information about the toolchain to load see: Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • pythonVERSION-numpy-gnu-hpc

  • pythonVERSION-numpy-gnu-hpc-devel

7.3.2 SciPy Python Library

SciPy is a collection of mathematical algorithms and convenience functions built on the NumPy extension of Python. It provides high-level commands and classes for manipulating and visualizing data. With SciPy, an interactive Python session becomes a data-processing and system-prototyping environment.

This package is available both for Python 2 (up to version 1.2.0 only) and 3. The specific compiler toolchain modules must be loaded for this library. The correct library module for the Python version used must be specified when loading this module. To load this module, run the following command:

> module load TOOLCHAIN pythonVERSION-scipy

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • pythonPYTHON_VERSION-scipy-gnu-hpc

  • pythonPYTHON_VERSION-scipy-gnu-hpc-devel

7.3.3 memkind — heap manager for heterogeneous memory platforms and mixed memory policies

The memkind library is a user-extensible heap manager built on top of jemalloc. It enables control over memory characteristics and a partitioning of the heap between kinds of memory. The kinds of memory are defined by operating system memory policies that have been applied to virtual address ranges. Memory characteristics supported by memkind without user extension include control of NUMA and page size features.

For more information, see:

Note

This tool is only available for AMD64/Intel 64.

7.3.4 Support for PMIx in Slurm and MPI libraries

PMIx abstracts the internals of MPI implementations for workload managers and unifies the way MPI jobs are started by the workload manager. With PMIx, there is no need to use the individual MPI launchers on Slurm, because srun will take care of this. In addition, the workload manager can determine the topology of the cluster, so you do not need to specify topologies manually.

7.3.5 OpenBLAS library — optimized BLAS library

OpenBLAS is an optimized BLAS (Basic Linear Algebra Subprograms) library based on GotoBLAS2 1.3, BSD version. It provides the BLAS API. It is shipped as a package enabled for environment modules, so it requires using Lmod to select a version. There are two variants of this library: an OpenMP-enabled variant, and a pthreads variant.

OpenMP-Enabled Variant

The OpenMP variant covers the following use cases:

  • Programs using OpenMP. This requires the OpenMP-enabled library version to function correctly.

  • Programs using pthreads. This requires an OpenBLAS library without pthread support. This can be achieved with the OpenMP-version. We recommend limiting the number of threads that are used to 1 by setting the environment variable OMP_NUM_THREADS=1.

  • Programs without pthreads and without OpenMP. Such programs can still take advantage of the OpenMP optimization in the library by linking against the OpenMP variant of the library.

When linking statically, ensure that libgomp.a is included by adding the linker flag -lgomp.

pthreads Variant

The pthreads variant of the OpenBLAS library can improve the performance of single-threaded programs. The number of threads used can be controlled with the environment variable OPENBLAS_NUM_THREADS.

Installation and Usage

This module requires loading a compiler toolchain beforehand. To select the latest version of this module provided, run the following command:

  • Standard version:

    > module load TOOLCHAIN openblas
  • OpenMP/pthreads version:

    > module load TOOLCHAIN openblas-pthreads

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • libopenblas-gnu-hpc

  • libopenblas-gnu-hpc-devel

  • libopenblas-pthreads-gnu-hpc

  • libopenblas-pthreads-gnu-hpc-devel

7.4 File format libraries

7.4.1 HDF5 HPC library — model, library, and file format for storing and managing data

HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of data types, and is designed for flexible and efficient I/O and for high-volume and complex data. HDF5 is portable and extensible, allowing applications to evolve in their use of HDF5.

There are serial and MPI variants of this library available. All flavors require loading a compiler toolchain module beforehand. The MPI variants also require loading the correct MPI flavor module.

To load the highest available serial version of this module, run the following command:

> module load TOOLCHAIN hdf5

When an MPI flavor is loaded, you can load the MPI version of this module by running the following command:

> module load TOOLCHAIN MPI_FLAVOR phdf5

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information about available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • hdf5-hpc-examples

  • hdf5-gnu-hpc-devel

  • libhdf5-gnu-hpc

  • libhdf5_cpp-gnu-hpc

  • libhdf5_fortran-gnu-hpc

  • libhdf5_hl_cpp-gnu-hpc

  • libhdf5_hl_fortran-gnu-hpc

  • hdf5-gnu-MPI_FLAVOR-hpc-devel

  • libhdf5-gnu-MPI_FLAVOR-hpc

  • libhdf5_fortran-gnu-MPI_FLAVOR-hpc

  • libhdf5_hl_fortran-MPI_FLAVOR-hpc

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

For general information about Lmod and modules, see Section 7.1, “Lmod — Lua-based environment modules”.

7.5 MPI libraries

Three different implementation of the Message Passing Interface (MPI) standard are provided standard with the HPC module:

  • Open MPI (version 3 and version 4)

  • MVAPICH2

  • MPICH

These packages have been built with full environment module support (LMOD).

The following packages are available:

  • For Open MPI:

    • user programs: openmpi3-gnu-hpc and openmpi4-gnu-hpc

    • shared libraries: libopenmpi3-gnu-hpc and libopenmpi4-gnu-hpc

    • development libraries, headers and tools required for building: openmpi3-gnu-hpc-devel and openmpi4-gnu-hpc-devel

    • documentation: openmpi3-gnu-hpc-docs and openmpi4-gnu-hpc-docs.

  • For MVAPICH2

    • user programs and libraries: mvapich2-gnu-hpc

    • development libraries, headers and tools for building: mvapich2-gnu-hpc-devel

    • documentation: mvapich2-gnu-hpc-doc

    For MPICH:

    • user programs and libraries: mpich-gnu-hpc

    • development libraries, headers and tools for building: mpich-gnu-hpc-devel

The different MPI implementations and versions are independent of each other, and can be installed in parallel.

Use environment modules to pick the version to use:

  • For Open MPI v.3:

    > module load TOOLCHAIN openmpi/3
  • For Open MPI v.4:

    > module load TOOLCHAIN openmpi/4
  • For MVAPICH2:

    > module load TOOLCHAIN mvapich2
  • For MPICH:

    > module load TOOLCHAIN mpich

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

7.6 Profiling and benchmarking libraries and tools

SUSE Linux Enterprise High Performance Computing provides tools for profiling MPI applications and benchmarking MPI performance.

7.6.1 IMB — Intel* MPI benchmarks

The Intel* MPI Benchmarks package provides a set of elementary benchmarks that conform to the MPI-1, MPI-2, and MPI-3 standards. You can run all of the supported benchmarks, or a subset specified in the command line, using a single executable file. Use command line parameters to specify various settings, such as time measurement, message lengths, and selection of communicators. For details, see the Intel* MPI Benchmarks User's Guide: https://software.intel.com/en-us/imb-user-guide.

For the IMB binaries to be found, a compiler toolchain and an MPI flavor must be loaded beforehand. To load this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR imb

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

  • imb-gnu-MPI_FLAVOR-hpc

7.6.2 PAPI HPC library — consistent interface for hardware performance counters

PAPI provides a tool with a consistent interface and methodology for the performance counter hardware found in most major microprocessors.

This package works with all compiler toolchains and does not require a compiler toolchain to be selected. Load the latest version provided by running the following command:

> module load TOOLCHAIN papi

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”.

List of master packages:

  • papi-hpc

  • papi-hpc-devel

For general information about Lmod and modules, see Section 7.1, “Lmod — Lua-based environment modules”.

7.6.3 mpiP — lightweight MPI profiling library

mpiP is a lightweight profiling library for MPI applications. Because it only collects statistical information about MPI functions, mpiP generates considerably less overhead and much less data than tracing tools. All the information captured by mpiP is task-local. It only uses communication during report generation, typically at the end of the experiment, to merge results from all of the tasks into one output file.

For this library a compiler toolchain and MPI flavor must be loaded beforehand. To load this module, run the following command:

> module load TOOLCHAIN MPI_FLAVOR mpip

For information about the toolchain to load, see Section 7.2, “GNU Compiler Toolchain Collection for HPC”. For information on available MPI flavors, see Section 7.5, “MPI libraries”.

List of master packages:

  • mpiP-gnu-MPI_FLAVOR-hpc

  • mpiP-gnu-MPI_FLAVOR-hpc-devel

  • mpiP-gnu-MPI_FLAVOR-hpc-doc

MPI_FLAVOR must be one of the supported MPI flavors described in Section 7.5, “MPI libraries”.

7.7 Creating environment containers with Singularity

You can deploy environments with preconfigured environment variables by using environment containers. Environment containers include only the components that are part of the environment, plus any required user applications. To create a container from the current HPC environment, use the container platform Singularity. Singularity is available from SUSE Package Hub. You can also use Spack to configure the environment to use with Singularity.

For more information, see the following documentation: