Release Notes #

spack checksum can automatically add new versions to package

new command: spack pkg grep to easily search package files

New maintainers directive

Add spack buildcache push (alias to buildcache create)

Allow using -j to control the parallelism of concretization

Add --exclude option to `spack external find'

editing: add higher-precedence SPACK_EDITOR environment variable

Many YAML formatting improvements from updating ruamel.yaml to the latest version supporting Python 3.6. .

Requirements and preferences should not define (non-git) versions

Environments now store spack version/commit in spack.lock

User can specify the name of the packages subdirectory in repositories

Add container images supporting RHEL alternatives

make version(…) kwargs explicit

buildcache create: reproducible tarballs

Bootstrap most of Spack dependencies using environments

Split satisfies(…​, strict=True/False) into two functions

spack install: simplify behavior when inside environments

Module file generation is disabled by default; you’ll need to enable it to use it

Support for Python 2 was deprecated in v0.19.0 and has been removed. v0.20.0 only supports Python 3.6 and higher.

Deprecated target names are no longer recognized by Spack. Use generic names instead:

blacklist and whitelist in module configuration were deprecated in v0.19.0 and are removed in this release. Use exclude and include instead.

The ignore= parameter of the extends() directive has been removed. It was not used by any builtin packages and is no longer needed to avoid conflicts in environment views.

Support for the old YAML buildcache format has been removed. It was deprecated in v0.19.0.

spack find --bootstrap has been removed. It was deprecated in v0.19.0. Use spack --bootstrap find instead.

spack bootstrap trust and spack bootstrap untrust are now removed, having been deprecated in v0.19.0. Use spack bootstrap enable and spack bootstrap disable.

The --mirror-name, --mirror-url, and --directory options to buildcache and mirror commands were deprecated in v0.19.0 and have now been removed. They have been replaced by positional arguments.

Deprecate env: as top level environment key

deprecate buildcache create –rel, buildcache install –allow-root

Support for very old perl-like spec format strings (e.g., $_$@$%@+$+$=) has been removed. This was deprecated in in v0.15.

bugfix: don’t fetch package metadata for unknown concrete specs

Improve package source code context display on error

Relax environment manifest filename requirements and lockfile identification criteria

installer.py: drop build edges of installed packages by default

Bugfix: package requirements with git commits

Package requirements: allow single specs in requirement lists

conditional variant values: allow boolean

spack uninstall: follow run/link edges on –dependents

Spec splicing To make binary installation more seamless in Spack, v0.23 introduces “splicing”, which allows users to deploy binaries using local, optimized versions of a binary interface, even if they were not built with that interface. For example, this would allow you to build binaries in the cloud using mpich and install them on a system using a local, optimized version of mvapich2 without rebuilding. Spack preserves full provenance for the installed packages and knows that they were built one way but deployed another. The intent is to leverage this across many key HPC binary packages, e.g. MPI, CUDA, ROCm, and libfabric. Fundamentally, splicing allows Spack to redeploy an existing spec with different dependencies than how it was built. There are two interfaces to splicing.

. Explicit Splicing In the concretizer config, you can specify a target spec and a replacement by hash.

concretizer:
  splice:
    explicit:
    - target: mpi
      replacement: mpich/abcdef

Here, every installation that would normally use the target spec will instead use its replacement. Above, any spec using any mpi will be spliced to depend on the specific mpich installation requested. This can go wrong if you try to replace something built with, e.g., openmpi with mpich, and it is on the user to ensure ABI compatibility between target and replacement specs. This currently requires some expertise to use, but it will allow users to reuse the binaries they create across more machines and environments. . Automatic Splicing (experimental) In the concretizer config, enable automatic splicing:

  concretizer:
    splice:
      automatic: true

or run:

spack config add concretizer:splice:automatic:true

The concretizer will select splices for ABI compatibility to maximize package reuse. Packages can denote ABI compatibility using the can_splice directive. No packages in Spack yet use this directive, so if you want to use this feature you will need to add can_splice annotations to your packages. We are working on ways to add more ABI compatibility information to the Spack package repository, and this directive may change in the future.

Further documentation: https://spack.readthedocs.io/en/latest/build_settings.html#splicing https://spack.readthedocs.io/en/latest/packaging_guide.html#specifying-abi-compatibility

Broader variant propagation You can specify propagated variants like hdf5 build_type==RelWithDebInfo or trilinos ++openmp to propagate a variant to all dependencies for which it is relevant. This is valid even if the variant does not exist on the package or its dependencies. See https://spack.readthedocs.io/en/latest/basic_usage.html#variants.

Query specs by namespace Allow a package’s namespace (indicating the repository it came from) to be treated like a variant. You can request packages from particular repos like this:

spack find zlib namespace=builtin
spack find zlib namespace=myrepo

Previously, the spec syntax only allowed namespaces to be prefixes of spec names, e.g. builtin.zlib. The previous syntax still works.

spack spec respects environment settings and unify:true spack spec did not previously respect environment lockfiles or unification settings, which made it difficult to see exactly how a spec would concretize within an environment. Now it does, so the output you get with spack spec will be the same as what your environment will concretize to when you run spack concretize. Similarly, if you provide multiple specs on the command line with spack spec, it will concretize them together if unify:true is set.

Less noisy spack spec output spack spec previously showed output like this:

> spack spec /v5fn6xo
Input spec
--------------------------------
 -   /v5fn6xo

 Concretized
--------------------------------
[+]  openssl@3.3.1%apple-clang@16.0.0~docs+shared arch=darwin-sequoia-m1
...

But the input spec is redundant, and we know we run spack spec to concretize the input spec. spack spec now only shows the concretized spec.

Better output for spack find -c In an environment, spack find -c lets you search the concretized, but not yet installed, specs, just as you would the installed ones. As with spack spec, this should make it easier for you to see what will be built before building and installing it.

spack -C <env>: use an environment’s configuration without activation Spack environments allow you to associate:

You can now also pass an environment to spack -C to use only the environment’s configuration, but not the specs or lockfile.

The new spack env track command takes a non-managed Spack environment and adds a symlink to Spack’s $environments_root directory, so that it will be included for reference counting for commands like spack uninstall and spack gc. If you use free-standing directory environments, this is useful for preventing Spack from removing things required by your environments. You can undo this tracking with the spack env untrack command.

Add -t short option for spack --backtrace spack -d / --debug enables backtraces on error, but it can be very verbose, and sometimes you just want the backtrace. spack -t / --backtrace provides that option.

gc: restrict to specific specs If you only want to garbage-collect specific packages, you can now provide them on the command line. This gives users finer-grained control over what is uninstalled.

oci buildcaches now support --only=package. You can now push just a package and not its dependencies to an OCI registry. This allows dependents of non-redistributable specs to be stored in OCI registries without an error.

Externals no longer override the preferred provider. External definitions could interfere with package preferences. Now, if openmpi is the preferred mpi, and an external mpich is defined, a new openmpi will be built if building it is possible. Previously we would prefer mpich despite the preference.

Composable cflags. This release fixes a longstanding bug that concretization would fail if there were different cflags specified in packages.yaml, compilers.yaml, or on the CLI. Flags and their ordering are now tracked in the concretizer and flags from multiple sources will be merged.

Fix concretizer Unification for included environments.

The old concretizer has been removed from Spack, along with the config:concretizer config option. Spack will emit a warning if the option is present in user configuration, since it now has no effect. Spack now uses a simpler bootstrapping mechanism, where a JSON prototype is tweaked slightly to get an initial concrete spec to download.

Best-effort expansion of spec matrices has been removed. This feature did not work with the “new” ASP-based concretizer, and did not work with unify: True or unify: when_possible. Use the exclude key for the environment to exclude invalid components, or use multiple spec matrices to combine the list of specs for which the constraint is valid and the list of specs for which it is not.

The old Cray platform (based on Cray PE modules) has been removed, and platform=cray is no longer supported. Since v0.19, Spack has handled Cray machines like Linux clusters with extra packages, and we have encouraged using this option to support Cray. The new approach allows us to correctly handle Cray machines with non-SLES operating systems, and it is much more reliable than making assumptions about Cray modules. See the v0.19 release notes for more details.

The config:install_missing_compilers config option has been deprecated, and it is a no-op when set in v0.23. Our new compiler dependency model will replace it with a much more reliable and robust mechanism in v1.0.

Config options that deprecated in v0.21 have been removed in v0.23. You can now only specify preferences for compilers, targets, and providers globally via the packages:all: section. Similarly, you can only specify versions: locally for a specific package.

Spack’s old test interface has been removed, having been deprecated in v0.22.0. All builtin packages have been updated to use the new interface. See the stand-alone test documentation

The spack versions --safe-only option, deprecated since v0.21.0, has been removed.

The --dependencies and --optimize arguments to spack ci have been deprecated.

Public binary caches now include an ML stack for Linux/aarch64. We now build an ML stack for Linux/aarch64 for all pull requests and on develop. The ML stack includes both CPU-only and CUDA builds for Horovod, Hugging Face, JAX, Keras, PyTorch, scikit-learn, TensorBoard, and TensorFlow, and related packages. The CPU-only stack also includes XGBoost. See https://cache.spack.io/tag/develop/?stack=ml-linux-aarch64-cuda.

archspec has been updated to v0.2.5, with support for zen5.

Spack’s CUDA package now supports the Grace Hopper 9.0a compute capability.

Bugfix: spack find -x in environments.

Spec splices are now robust to duplicate nodes with the same name in a spec.

Cache per-compiler libc calculations for performance.

Fixed a bug in external detection for openmpi.

Mirror configuration allows username/password as environment variables.

Default library search caps maximum depth/

Unify interface for spack spec and spack solve commands.

Spack no longer RPATHs directories in the default library search path.

Improved performance of Spack database.

Enable package reuse for packages with versions from git refs.

Improved tracking of task queueing/requeueing in the installer.

Forward compatibility with Spack 0.23 packages with language dependencies.

Forward compatibility with urllib from Python 3.12.6+.

Bump vendored archspec for better aarch64 support.

Fix regression in \\{variants.X} and \\{variants.X.value} format strings.

Ensure shell escaping of environment variable values in load and activate commands.

Fix an issue where spec[pkg] considers specs outside the current DAG.

Do not halt concretization on unknown variants in externals.

Improve validation of develop config section/

Explicitly disable ccache if turned off in config, to avoid cache pollution.

Improve backwards compatibility in include_concrete.

Fix issue where package tags were sometimes repeated.

Make setup-env.sh “sourced only” by dropping execution bits.

Make certain source/binary fetch errors recoverable instead of a hard error.

Remove debug statements in package hash computation.

Remove redundant clingo warnings.

Remove hard-coded layout version.

Do not initialize previous store state in use_store.

chapel major update/

Fix reuse of externals on Linux.

Ensure parent gcc-runtime version >= child.

Ensure the latest gcc-runtime is rpath’ed when multiple exist among link deps.

Improve version detection of glibc.

Improve heuristics for solver.

Make strong preferences override reuse.

Reduce verbosity when C compiler is missing.

Make missing ccache executable an error when required.

Make every environment view containing python a venv.

Fix external detection for compilers with os but no target.

Fix version optimization for roots.

Handle common implementations of pagination of tags in OCI build caches.

Apply fetched patches to develop specs.

Avoid Windows wrappers for filesystem utilities on non-Windows.

Fix formatting issue in spack audit.

Require libiconv for iconv. Notice that glibc/musl also provide iconv, but are not guaranteed to be complete. Set packages:iconv:require:[glibc] to restore the old behavior.

protobuf: fix 3.4:3.21 patch checksum.

protobuf: update hash for patch needed when=“@3.4:3.21”.

git: bump v2.39 to 2.45; deprecate unsafe versions.

gcc: use -rpath \{rpath_dir} not -rpath=\\{rpath dir}.

Remove mesa18 and libosmesa.

Enforce consistency of gl providers.

py-matplotlib: qualify when to do a post install.

rust: fix v1.78.0 instructions.

suite-sparse: improve setting of the libs property.

netlib-lapack: provide blas and lapack together.

Add report AccountUtilizationByQOS to sreport.

AccountUtilizationByUser is able to be filtered by QOS.

Add autodetected gpus to the output of slurmd -C

Add ability to submit jobs with multiple QOS. These are sorted by priority highest being the first.

Removed the instant on feature from switch/hpe_slingshot.

slurmctld - Changed incoming RPC handling to dedicated thread pool with asynchronous handling of I/O that can be configured via conmgr_* entries under SlurmctldParameters in slurm.conf.

Add SchedulerParameters=bf_allow_magnetic_slot option. It allows jobs in magnetic reservations to be planned by backfill scheduler.

Add TopologyParam=TopoMaxSizeUnroll=# to allow --nodes=<min>-<max> for topology/block.

Add DataParserParameters slurm.conf parameter to allow setting default value for CLI --json and --yaml arguments.

Hardware collectives in switch/hpe_slingshot now requires enable_stepmgr.

Added connection related parameters to slurm.conf under SlurmctldParameters:
conmgr_max_connections: Defaults to 150 connections.

conmgr_threads: Defaults to 64 threads for slurmctld.

conmgr_use_poll: Defaults is to use epoll in Linux.

conmgr_connect_timeout: Defaults to MessageTimeout.

conmgr_read_timeout: Defaults to MessageTimeout.

conmgr_wait_write_delay: Defaults to MessageTimeout.

conmgr_write_timeout: Defaults to MessageTimeout.

Add SlurmctldParamters=ignore_constraint_validation to ignore constraint/feature validation at submission.

Add SchedulerParameters=bf_topopt_enable option to enable experimental hook to control backfill.

Remove srun --cpu-bind=rank.

Add "%b" as a file name pattern for the array task id modulo 10.

sacct - Respect --noheader for --batch-script and --env-vars.

Add sacctmgr ping command to query status of slurmdbd.

sbcast - Add --nodelist option to specify where files are transmitted to

sbcast - Add --no-allocation option to transmit files to nodes outside of a job allocation.

slurmdbd - Add -u option. This is used to determine if restarting the DBD will result in database conversion.

Remove salloc --get-user-env.

scontrol - Add --json/--yaml support to listpids.

scontrol - Add liststeps.

scontrol - Add listjobs.

scontrol show topo - Show aggregated block sizes when using topology/block.

Remove burst_buffer/lua call slurm.job_info_to_string().

job_submit/lua - Add assoc_qos attribute to job_desc to display all potential QOS’s for a job’s association.

job_submit/lua - Add slurm.get_qos_priority() function to retrieve the given QOS’s priority.

Removed fields deprecated in the Slurm-23.11 release from v0.0.42 endpoints.

Removed v0.0.39 plugins.

Set data_parser/v0.0.42+prefer_refs flag to default.

Add data_parser/v0.0.42+minimize_refs flag to inline single referenced schemas in the OpenAPI schema to get default behavior of data_parser/v0.0.41.

Rename v0.0.42 JOB_INFO field minimum_switches to required_switches to reflect the actual behavior.

Rename v0.0.42 ACCOUNT_CONDITION field assocation to association (typo).

Tag slurmdb/v0.0.42/jobs pid field deprecated.

With client commands MIN_MEMORY will show mem_per_tres if specified.

Fix errno message about bad constraint.

slurmctld - Fix crash and possible split brain issue if the backup controller handles an scontrol reconfigure while in control before the primary resumes operation.

Fix stepmgr not getting dynamic node addrs from the controller.

stepmgr - avoid Unexpected missing socket errors.

Fix scontrol show steps with dynamic stepmgr.

Deny jobs using the R: option of --signal if PreemptMode=OFF globally.

Force jobs using the R: option of --signal to be preemptable by requeue or cancel only. If PreemptMode on the partition or QOS is off or suspend, the job will default to using PreemptMode=cancel.

If --mem-per-cpu exceeds MaxMemPerCPU, the number of CPUs per task will always be increased even if --cpus-per-task was specified. This is needed to ensure each task gets the expected amount of memory.

Fix compilation issue on openSUSE Leap 15.

Fix jobs using more nodes than needed when not using -N.

Fix issue with allocation being allocated less resources than needed when using --gres-flags=enforce-binding.

select/cons_tres - Fix errors with MaxCpusPerSocket partition limit. Used CPUs/cores weren’t counted properly, nor limiting free ones to avail, when the socket was partially allocated, or the job request went beyond this limit.

Fix issue when jobs were preempted for licenses even if there were enough licenses available.

Fix srun ntasks calculation inside an allocation when nodes are requested using a min-max range.

Print correct number of digits for TmpDisk in sdiag.

Fix a regression in 24.11 which caused file transfers to a job with sbcast to not join the job container namespace.

data_parser/v0.0.40 - Prevent a segfault in the slurmrestd when dumping data with v0.0.40+complex data parser.

Remove logic to force lowercase GRES names.

data_parser/v0.0.42 - Prevent the association id from always being dumped as NULL when parsing in complex mode. Instead it will now dump the id. This affects the following endpoints:

slurmctld - Add new RPC rate limiting feature. This is enabled through SlurmctldParameters=rl_enable, otherwise disabled by default.

Make scontrol reconfigure and sending a SIGHUP to the slurmctld behave the same. If you were using SIGHUP as a 'lighter' scontrol reconfigure to rotate logs please update your scripts to use SIGUSR2 instead.

Change cloud nodes to show by default. PrivateData=cloud is no longer needed.

sreport - Count planned (FKA reserved) time for jobs running in IGNORE_JOBS reservations. Previously was lumped into IDLE time.

job_container/tmpfs - Support running with an arbitrary list of private mount points (/tmp and /dev/shm are the default, but not required).

job_container/tmpfs - Set more environment variables in InitScript.

Make all cgroup directories created by Slurm owned by root. This was the behavior in cgroup/v2 but not in cgroup/v1 where by default the step directories ownership were set to the user and group of the job.

accounting_storage/mysql - change purge/archive to calculate record ages based on end time, rather than start or submission times.

job_submit/lua - add support for log_user() from slurm_job_modify().

Run the following scripts in slurmscriptd instead of slurmctld: ResumeProgram, ResumeFailProgram, SuspendProgram, ResvProlog, ResvEpilog, and RebootProgram (only with SlurmctldParameters=reboot_from_controller).

Only permit changing log levels with srun --slurmd-debug by root or SlurmUser.

slurmctld will fatal() when reconfiguring the job_submit plugin fails.

Add PowerDownOnIdle partition option to power down nodes after nodes become idle.

Add “[jobid.stepid]” prefix from slurmstepd and “slurmscriptd” prefix from slurmcriptd to Syslog logging. Previously was only happening when logging to a file.

Add purge and archive functionality for job environment and job batch script records.

Extend support for Include files to all "configless" client commands.

Make node weight usable for powered down and rebooting nodes.

Removed “launch” plugin.

Add “Extra” field to job to store extra information other than a comment.

Add usage gathering for AMD (requires ROCM 5.5+) and NVIDIA gpus.

Add job’s allocated nodes, features, oversubscribe, partition, and reservation to SLURM_RESUME_FILE output for power saving.

Automatically create directories for stdout/stderr output files. Paths may use %j and related substitution characters as well.

Add --tres-per-task to salloc/sbatch/srun.

Allow nodefeatures plugin features to work with cloud nodes. e.g. - Powered down nodes have no active changeable features.

Make slurmstepd cgroups constrained by total configured memory from slurm.conf (NodeName=<> RealMemory=#) instead of total physical memory.

node_features/helpers - add support for the OR and parentheses operators in a --constraint expression.

slurmctld will fatal() when [Prolog|Epilog]Slurmctld are defined but are not executable.

Validate node registered active features are a super set of node’s currently active changeable features.

On clusters without any PrologFlags options, batch jobs with failed prologs nolonger generate an output file.

Add SLURM_JOB_START_TIME and SLURM_JOB_END_TIME environment variables.

Add SuspendExcStates option to slurm.conf to avoid suspending/powering down specific node states.

Add support for DCMI power readings in IPMI plugin.

slurmrestd served /slurm/v0.0.39 and /slurmdb/v0.0.39 endpoints had major changes from prior versions. Almost all schemas have been renamed and modified. Sites using OpenAPI Generator clients are highly suggested to upgrade to to using atleast version 6.x due to limitations with prior versions.

Allow for --nodelist to contain more nodes than required by --nodes.

Rename “nodes” to “nodes_resume” in SLURM_RESUME_FILE job output.

Rename “all_nodes” to “all_nodes_resume” in SLURM_RESUME_FILE output.

Add jobcomp/kafka plugin.

Add new PreemptParameters=reclaim_licenses option which will allow higher priority jobs to preempt jobs to free up used licenses. (This is only enabled for with PreemptModes of CANCEL and REQUEUE, as Slurm cannot guarantee suspended jobs will release licenses correctly.)

hpe/slingshot - add support for the instant-on feature.

Add ability to update SuspendExc* parameters with scontrol.

Add ability to restore SuspendExc* parameters on restart with slurmctld -R option.

Add ability to clear a GRES specification by setting it to "0" via “scontrol update job”.

Add SLURM_JOB_OVERSUBSCRIBE environment variable for Epilog, Prolog, EpilogSlurmctld, PrologSlurmctld, and mail ouput.

System node down reasons are appended to existing reasons, separated by ':'.

New command scrun has been added. scrun acts as an Open Container Initiative (OCI) runtime proxy to run containers seamlessly via Slurm.

Fixed GpuFreqDef option. When set in slurm.conf, it will be used if --gpu-freq was not explicitly set by the job step.

job_container.conf - Added “Dirs” option to list desired private mount points.

node_features plugins - invalid users specified for AllowUserBoot will now result in fatal() rather than just an error.

Deprecate AllowedKmemSpace, ConstrainKmemSpace, MaxKmemPercent, and MinKmemSpace.

Allow jobs to queue even if the user is not in AllowGroups when EnforcePartLimits=no is set. This ensures consistency for all the Partition access controls, and matches the documented behavior for EnforcePartLimits.

Add InfluxDBTimeout parameter to acct_gather.conf.

job_container/tmpfs - add support for expanding %h and %n in BasePath.

slurm.conf - Removed SlurmctldPlugstack option.

Add new SlurmctldParameters=validate_nodeaddr_threads=<number> option to allow concurrent hostname resolution at slurmctld startup.

Add new AccountingStoreFlags=job_extra option to store a job’s extra field in the database.

Add new “defer_batch” option to SchedulerParameters to only defer scheduling for batch jobs.

Add new DebugFlags option “JobComp” to replace “Elasticsearch”.

Add configurable job requeue limit parameter - MaxBatchRequeue - in slurm.conf to permit changes from the old hard-coded value of 5.

helpers.conf - Allow specification of node specific features.

helpers.conf - Allow many features to one helper script.

job_container/tmpfs - Add “Shared” option to support shared namespaces. This allows autofs to work with the job_container/tmpfs plugin when enabled.

acct_gather.conf - Added EnergyIPMIPowerSensors=Node=DCMI and Node=DCMI_ENHANCED.

Add new “getnameinfo_cache_timeout=<number>” option to CommunicationParameters to adjust or disable caching the results of getnameinfo().

Add new PrologFlags=ForceRequeueOnFail option to automatically requeue batch jobs on Prolog failures regardless of the job --requeue setting.

Add HealthCheckNodeState=NONDRAINED_IDLE option.

Add “explicit” to Flags in gres.conf. This makes it so the gres is not automatically added to a job’s allocation when --exclusive is used. Note that this is a per-node flag.

Moved the “preempt_” options from SchedulerParameters to PreemptParameters, and dropped the prefix from the option names. (The old options will still be parsed for backwards compatibility, but are now undocumented.)

Add LaunchParameters=ulimit_pam_adopt, which enables setting RLIMIT_RSS in adopted processes.

Update SwitchParameters=job_vni to enable/disable creating job VNIs for all jobs, or when a user requests them.

Update SwitchParameters=single_node_vni to enable/disable creating single node VNIs for all jobs, or when a user requests them.

Add ability to preserve SuspendExc* parameters on reconfig with ReconfigFlags=KeepPowerSaveSettings.

slurmdbd.conf - Add new AllResourcesAbsolute to force all new resources to be created with the Absolute flag.

topology/tree - Add new TopologyParam=SwitchAsNodeRank option to reorder nodes based on switch layout. This can be useful if the naming convention for the nodes does not natually map to the network topology.

Removed the default setting for GpuFreqDef. If unset, no attempt to change the GPU frequency will be made if --gpu-freq is not set for the step.

sacctmgr - no longer force updates to the AdminComment, Comment, or SystemComment to lower-case.

sinfo - Add -F/--future option to sinfo to display future nodes.

sacct - Rename “Reserved” field to “Planned” to match sreport and the nomenclature of the 'Planned' node.

scontrol - advanced reservation flag MAINT will no longer replace nodes, similar to STATIC_ALLOC

sbatch - add parsing for #PBS -d and #PBS -w.

scontrol show assoc_mgr will show username(uid) instead of uid in QoS section.

Add strigger --draining and -R/--resume options.

Change --oversubscribe and --exclusive to be mutually exclusive for job submission. Job submission commands will now fatal if both are set. Previously, these options would override each other, with the last one in the job submission command taking effect.

scontrol - Requested TRES and allocated TRES will now always be printed when showing jobs, instead of one TRES output that was either the requested or allocated.

srun --ntasks-per-core now applies to job and step allocations. Now, use of --ntasks-per-core=1 implies --cpu-bind=cores and --ntasks-per-core>1 implies --cpu-bind=threads.

salloc/sbatch/srun - Check and abort if ntasks-per-core > threads-per-core.

scontrol - Add ResumeAfter=<secs> option to “scontrol update nodename=”.

Add a new “nodes=” argument to scontrol setdebug to allow the debug level on the slurmd processes to be temporarily altered.

Add a new “nodes=” argument to “scontrol setdebugflags” as well.

Make it so scrontab prints client-side the job_submit() error messsage (which can be set i.e. by using the log_user() function for the lua plugin).

scontrol - Reservations will not be allowed to have STATIC_ALLOC or MAINT flags and REPLACE[_DOWN] flags simultaneously.

scontrol - Reservations will only accept one reoccurring flag when being created or updated.

scontrol - A reservation cannot be updated to be reoccurring if it is already a floating reservation.

squeue - removed unused “%s” and “SelectJobInfo” formats.

squeue - align print format for exit and derived codes with that of other components (<exit_status>:<signal_number>).

sacct - Add --array option to expand job arrays and display array tasks on separate lines.

Partial support for “--json” and “--yaml” formated outputs have been implemented for sacctmgr, sdiag, sinfo, squeue, and scontrol. The resultant data ouput will be filtered by normal command arguments. Formatting arguments will continue to be ignored.

salloc/sbatch/srun - extended the --nodes syntax to allow for a list of valid node counts to be allocated to the job. This also supports a "step count" value (e.g., --nodes=20-100:20 is equivalent to --nodes=20,40,60,80,100) which can simplify the syntax when the job needs to scale by a certain "chunk" size.

srun - add user requestible vnis with “--network=job_vni” option.

srun - add user requestible single node VNIs with the “--network=single_node_vni” option.

job_container plugins - container_p_stepd_create() function signature replaced uint32_t uid with stepd_step_rec_t* step.

gres plugins - gres_g_get_devices() function signature replaced pid_t pid with stepd_step_rec_t* step.

cgroup plugins - task_cgroup_devices_constrain() function signature removed pid_t pid.

task plugins - replace task_p_pre_set_affinity(), task_p_set_affinity(), and task_p_post_set_affinity() with task_p_pre_launch_priv() like it was back in slurm 20.11.

Allow for concurrent processing of job_submit_g_submit() and job_submit_g_modify() calls. If your plugin is not capable of concurrent operation you must add additional locking within your plugin.

Removed return value from slurm_list_append().

The List and ListIterator types have been removed in favor of list_t and list_itr_t respectively.

burst buffer plugins - add bb_g_build_het_job_script(). bb_g_get_status() - added authenticated UID and GID. bb_g_run_script() - added job_info argument.

burst_buffer.lua - Pass UID and GID to most hooks. Pass job_info (detailed job information) to many hooks. See etc/burst_buffer.lua.example for a complete list of changes. WARNING: Backwards compatibility is broken for slurm_bb_get_status: UID and GID are passed before the variadic arguments. If UID and GID are not explicitly listed as arguments to slurm_bb_get_status(), then they will be included in the variadic arguments. Backwards compatibility is maintained for all other hooks because the new arguments are passed after the existing arguments.

node_features plugins - node_features_p_reboot_weight() function removed. node_features_p_job_valid() - added parameter feature_list. node_features_p_job_xlate() - added parameters feature_list and job_node_bitmap.

New data_parser interface with v0.0.39 plugin.

The template slurmrestd.service unit file now defaults to listen on both the Unix socket and the slurmrestd port.

The template slurmrestd.service unit file now defaults to enable auth/jwt and the munge unit is no longer a dependency by default.

Add extra “EnvironmentFile=-/etc/default/$service” setting to service files.

Allow jobs to pack onto nodes already rebooting with the desired features.

Reset job start time after nodes are rebooted, previously only done for cloud/power save boots.

Node features (if any) are passed to RebootProgram if run from slurmctld.

Fail srun when using invalid --cpu-bind options (e.g. --cpu-bind=map_cpu:99 when only 10 CPUs are allocated).

Storing batch scripts and env vars are now in indexed tables using substantially less disk space. Those storing scripts in 21.08 will all be moved and indexed automatically.

Run MailProg through slurmscriptd instead of directly fork+exec()'ing from slurmctld.

Add acct_gather_interconnect/sysfs plugin.

Future and Cloud nodes are treated as "Planned Down" in usage reports.

Add new shard plugin for sharing GPUs but not with mps.

Add support for Lenovo SD650 V2 in acct_gather_energy/xcc plugin.

Remove cgroup_allowed_devices_file.conf, since the default policy in modern kernels is to whitelist by default. Denying specific devices must be done through gres.conf.

Node state flags (DRAIN, FAILED, POWERING UP, etc.) will be cleared now if node state is updated to FUTURE.

srun will no longer read in SLURM_CPUS_PER_TASK. This means you will implicitly have to specify --cpus-per-task on your srun calls, or set the new SRUN_CPUS_PER_TASK environment variable to accomplish the same thing.

Remove connect_timeout and timeout options from JobCompParams as there’s no longer a connectivity check happening in the jobcomp/elasticsearch plugin when setting the location off of JobCompLoc.

Add support for hourly reoccurring reservations.

Allow nodes to be dynamically added and removed from the system. Configure MaxNodeCount to accomodate nodes created with dynamic node registrations (slurmd -Z --conf="") and scontrol.

Added support for Cgroup Version 2.

sacct - allocations made by srun will now always display the allocation and step(s). Previously, the allocation and step were combined when possible.

cons_tres - change definition of the "least loaded node" (LLN) to the node with the greatest ratio of available CPUs to total CPUs.

Add support to ship Include configuration files with configless.

Provide a detailed reason in the job log as to why it has been terminated when hitting a resource limit.

Pass and use alias_list through credential instead of environment variable.

Add ability to get host addresses from nss_slurm.

Enable reverse fanout for cloud+alias_list jobs.

Add support to delete/update nodes by specifying nodesets or the 'ALL' keyword alongside the delete/update node message nodelist expression (i.e. scontrol delete/update NodeName=ALL or scontrol delete/update NodeName=ns1,nodes[1-3]).

Expanded the set of environment variables accessible through Prolog/Epilog and PrologSlurmctld/EpilogSlurmctld to include SLURM_JOB_COMMENT, SLURM_JOB_STDERR, SLURM_JOB_STDIN, SLURM_JOB_STDOUT, SLURM_JOB_PARTITION, SLURM_JOB_ACCOUNT, SLURM_JOB_RESERVATION, SLURM_JOB_CONSTRAINTS, SLURM_JOB_NUM_HOSTS, SLURM_JOB_CPUS_PER_NODE, SLURM_JOB_NTASKS, and SLURM_JOB_RESTART_COUNT.

Attempt to requeue jobs terminated by slurm.conf changes (node vanish, node socket/core change, etc). Processes may still be running on excised nodes. Admin should take precautions when removing nodes that have jobs on running on them.

Add switch/hpe_slingshot plugin.

Add new SchedulerParameters option bf_licenses to track licenses as within the backfill scheduler.

AcctGatherEnergyType rsmi is now gpu.

TaskAffinity parameter was removed from cgroup.conf.

Fatal if the mutually-exclusive JobAcctGatherParams options of UsePss and NoShared are both defined.

KeepAliveTime has been moved into CommunicationParameters. The standalone option will be removed in a future version.

preempt/qos - add support for WITHIN mode to allow for preemption between jobs within the same QOS.

Fatal error if CgroupReleaseAgentDir is configured in cgroup.conf. The option has long been obsolete.

Fatal if more than one burst buffer plugin is configured.

Added keepaliveinterval and keepaliveprobes to CommunicationParameters.

Added new max_token_lifespan=<seconds> to AuthAltParameters to allow sites to restrict the lifespan of any requested ticket by an unprivileged user.

Disallow slurm.conf node configurations with NodeName=ALL.

Remove support for (non-functional) --cpu-bind=boards.

Added --prefer option at job submission to allow for 'soft' constraints.

Add condflags=open to sacctmgr show events to return open/currently down events.

sacct -f flag implies -c flag.

srun --overlap now allows the step to share all resources (CPUs, memory, and GRES), where previously --overlap only allowed the step to share CPUs with other steps.

openapi/v0.0.35 - Plugin has been removed.

burst_buffer plugins - err_msg added to bb_p_job_validate().

openapi - added flags to slurm_openapi_p_get_specification(). Existing plugins only need to update their prototype for the function as manipulating the flags pointer is optional.

openapi - Added OAS_FLAG_MANGLE_OPID to allow plugins to request that the operationId of path methods be mangled with the full path to ensure uniqueness.

openapi/[db]v0.0.36 - Plugins have been marked as deprecated and will be removed in the next major release.

switch plugins - add switch_g_job_complete() function.

Removed gres/mic plugin used to support Xeon Phi coprocessors.

Add LimitFactor to the QOS. A float that is factored into an associations GrpTRES limits. For example, if the LimitFactor is 2, then an association with a GrpTRES of 30 CPUs, would be allowed to allocate 60 CPUs when running under this QOS.

A job’s next_step_id counter now resets to 0 after being requeued. Previously, the step id’s would continue from the job’s last run.

API change: Removed slurm_kill_job_msg and modified the function signature for slurm_kill_job2. slurm_kill_job2 should be used instead of slurm_kill_job_msg.

AccountingStoreFlags=job_script allows you to store the job’s batch script.

AccountingStoreFlags=job_env allows you to store the job’s env vars.

Removed sched/hold plugin.

cli_filter/lua, jobcomp/lua, job_submit/lua now load their scripts from the same directory as the slurm.conf file (and thus now will respect changes to the SLURM_CONF environment variable).

SPANK - call slurm_spank_init if defined without slurm_spank_slurmd_exit in slurmd context.

Add new PLANNED state to a node to represent when the backfill scheduler has it planned to be used in the future instead of showing as IDLE. sreport also has changed it’s cluster utilization report column name from 'Reserved' to 'Planned' to match this nomenclature.

Put node into INVAL state upon registering with an invalid node configuration. Node must register with a valid configuration to continue.

Remove SLURM_DIST_LLLP environment variable in favor of just SLURM_DISTRIBUTION.

Make --cpu-bind=threads default for --threads-per-core — can be overridden by the CLI or an environment variable.

slurmd - allow multiple comma-separated controllers to be specified in configless mode with --conf-server

Manually powering down of nodes with scontrol now ignores SuspendExc<Nodes|Parts>.

Distinguish queued reboot requests (REBOOT@) from issued reboots (REBOOT^).

auth/jwt - add support for RS256 tokens. Also permit the username in the 'username' field in addition to the 'sun' (Slurm UserName) field.

service files - change dependency to network-online rather than just network to ensure DNS and other services are available.

Add "Extra" field to node to store extra information other than a comment.

Add ResumeTimeout, SuspendTimeout and SuspendTime to Partitions.

The memory.force_empty parameters is no longer set by jobacct_gather/cgroup when deleting the cgroup`. This previously caused a significant delay (~2s) when terminating a job, and is not believed to have provided any perceivable benefit. However, this may lead to slightly higher reported kernel mem page cache usage since the kernel cgroup memory is no longer freed immediately.

TaskPluginParam=verbose is now treated as a default. Previously it would be applied regardless of the job specifying a --cpu-bind.

Add node_reg_mem_percent SlurmctldParameter to define percentage of memory nodes are allowed to register with.

Define and separate node power state transitions. Previously a powering down node was in both states, POWERING_OFF and POWERED_OFF. These are now separated. e.g. IDLE+POWERED_OFF (IDLE~) → IDLE+POWERING_UP (IDLE#) - Manual power up or allocation → IDLE → IDLE+POWER_DOWN (IDLE!) - Node waiting for power down → IDLE+POWERING_DOWN (IDLE%) - Node powering down → IDLE+POWERED_OFF (IDLE~) - Powered off

Some node state flag names have changed. These would be noticeable for example if using a state flag to filter nodes with sinfo. e.g. POWER_UP → POWERING_UP POWER_DOWN → POWERED_DOWN POWER_DOWN now represents a node pending power down

Create a new process called slurmscriptd which runs PrologSlurmctld and EpilogSlurmctld. This avoids fork() calls from slurmctld, and can avoid performance issues if the slurmctld has a large memory footprint.

Pass JSON of job to node mappings to ResumeProgram.

QOS accrue limits only apply to the job QOS, not partition QOS.

Any return code from SPANK plugin or SPANK function that is not SLURM_SUCCESS (zero) will be considered to be an error. Previously, only negative return codes were considered an error.

Add support for automatically detecting and broadcasting executable shared object dependencies for sbcast and srun --bcast.

All SPANK error codes now start at 3000. Where previously SPANK would give a return code of 1, it will now return 3000. This change will break ABI compatibility with SPANK plugins compiled against older version of Slurm.

SPANK plugins are now required to match the current Slurm release, and must be recompiled for each new Slurm major release. (They do not need to be recompiled when upgrading between maintenance releases.)

SLURM_NODE_ALIASES now has brackets around the node’s address to be able to distinguish IPv6 addresses. e.g. <node_name>:[<node_addr>]:<node_hostname>

The job_container/tmpfs plugin now requires PrologFlags=contain to be set in slurm.conf.

Limit max_script_size to 512 MB.

Errors detected in the parser handlers due to invalid configurations are now propagated and can lead to fatal (and thus exit) the calling process.

Enforce a valid configuration for AccountingStorageEnforce in slurm.conf. If the configuration is invalid, then an error message will be printed and the command or daemon (including slurmctld) will not run.

Removed AccountingStoreJobComment option. Please update your config to use AccountingStoreFlags=job_comment instead.

Removed DefaultStorage{Host,Loc,Pass,Port,Type,User} options.

Removed CacheGroups, CheckpointType, JobCheckpointDir, MemLimitEnforce, SchedulerPort, SchedulerRootFilter options.

Added Script to DebugFlags for debugging slurmscriptd (the process that runs slurmctld scripts such as PrologSlurmctld and EpilogSlurmctld).

Rename SbcastParameters to BcastParameters.

systemd service files - add new “-s” option to each daemon which will change the working directory even with the -D option. (Ensures any core files are placed in an accessible location, rather than /.)

Added BcastParameters=send_libs and BcastExclude options.

Remove the (incomplete) burst_buffer/generic plugin.

Make SelectTypeParameters=CR_Core_Memory default for cons_tres and cons_res.

Remove support for TaskAffinity=yes in cgroup.conf. Adding task/affinity to TaskPlugins in slurm.conf is strongly recommended instead.

Changed the --format handling for negative field widths (left justified) to apply to the column headers as well as the printed fields.

Invalidate multiple partition requests when using partition based associations.

scrontab - create the temporary file under the TMPDIR environment variable (if set), otherwise continue to use TmpFS as configured in slurm.conf.

sbcast / srun --bcast - removed support for zlib compression. lz4 is vastly superior in performance, and (counter-intuitively) zlib could provide worse performance than no compression at all on many systems.

sacctmgr - changed column headings to ParentID and ParentName instead of Par ID and "Par Name` respectively.

SALLOC_THREADS_PER_CORE and SBATCH_THREADS_PER_CORE have been added as input environment variables for salloc and sbatch, respectively. They do the same thing as --threads-per-core.

Don’t display node’s comment with scontrol show nodes unless set.

Added SLURM_GPUS_ON_NODE environment variable within each job/step.

sreport - change to sorting TopUsage by the --tres option.

slurmrestd - do not run allow operation as SlurmUser/root by default.

scontrol show node now shows State as base_state+flags instead of shortened state with flags appended. eg. IDLE# → IDLE+POWERING_UP. Also POWER state flag string is POWERED_DOWN.

scrontab - add ability to update crontab from a file or standard input.

scrontab - added ability to set and expand variables.

Make srun sensitive to BcastParameters.

Added sbcast/srun --send-libs, sbcast --exclude and srun --bcast-exclude.

Changed ReqMem field in sacct to match memory from ReqTRES. It now shows the requested memory of the whole job with a letter appended indicating units (M for megabytes, G for gigabytes, etc.). ReqMem is only displayed for the job, since the step does not have requested TRES. Previously ReqMem was also displayed for the step but was just displaying ReqMem for the job.

jobcomp plugin: change plugin API to jobcomp_p_*().

sched plugin: change plugin API to sched_p_*() and remove slurm_sched_p_initial_priority() call.

step_ctx code has been removed from the api.

slurm_stepd_get_info()/stepd_get_info() has been removed from the api.

The v0.0.35 OpenAPI plugin has now been marked as deprecated. Please convert your requests to the v0.0.37 OpenAPI plugin.