Release Notes #

spack checksum can automatically add new versions to package

new command: spack pkg grep to easily search package files

New maintainers directive

Add spack buildcache push (alias to buildcache create)

Allow using -j to control the parallelism of concretization

Add --exclude option to `spack external find'

editing: add higher-precedence SPACK_EDITOR environment variable

Many YAML formatting improvements from updating ruamel.yaml to the latest version supporting Python 3.6. .

Requirements and preferences should not define (non-git) versions

Environments now store spack version/commit in spack.lock

User can specify the name of the packages subdirectory in repositories

Add container images supporting RHEL alternatives

make version(…) kwargs explicit

buildcache create: reproducible tarballs

Bootstrap most of Spack dependencies using environments

Split satisfies(…​, strict=True/False) into two functions

spack install: simplify behavior when inside environments

Module file generation is disabled by default; you’ll need to enable it to use it

Support for Python 2 was deprecated in v0.19.0 and has been removed. v0.20.0 only supports Python 3.6 and higher.

Deprecated target names are no longer recognized by Spack. Use generic names instead:

blacklist and whitelist in module configuration were deprecated in v0.19.0 and are removed in this release. Use exclude and include instead.

The ignore= parameter of the extends() directive has been removed. It was not used by any builtin packages and is no longer needed to avoid conflicts in environment views.

Support for the old YAML buildcache format has been removed. It was deprecated in v0.19.0.

spack find --bootstrap has been removed. It was deprecated in v0.19.0. Use spack --bootstrap find instead.

spack bootstrap trust and spack bootstrap untrust are now removed, having been deprecated in v0.19.0. Use spack bootstrap enable and spack bootstrap disable.

The --mirror-name, --mirror-url, and --directory options to buildcache and mirror commands were deprecated in v0.19.0 and have now been removed. They have been replaced by positional arguments.

Deprecate env: as top level environment key

deprecate buildcache create –rel, buildcache install –allow-root

Support for very old perl-like spec format strings (e.g., $_$@$%@+$+$=) has been removed. This was deprecated in in v0.15.

bugfix: don’t fetch package metadata for unknown concrete specs

Improve package source code context display on error

Relax environment manifest filename requirements and lockfile identification criteria

installer.py: drop build edges of installed packages by default

Bugfix: package requirements with git commits

Package requirements: allow single specs in requirement lists

conditional variant values: allow boolean

spack uninstall: follow run/link edges on –dependents

Spec splicing To make binary installation more seamless in Spack, v0.23 introduces “splicing”, which allows users to deploy binaries using local, optimized versions of a binary interface, even if they were not built with that interface. For example, this would allow you to build binaries in the cloud using mpich and install them on a system using a local, optimized version of mvapich2 without rebuilding. Spack preserves full provenance for the installed packages and knows that they were built one way but deployed another. The intent is to leverage this across many key HPC binary packages, e.g. MPI, CUDA, ROCm, and libfabric. Fundamentally, splicing allows Spack to redeploy an existing spec with different dependencies than how it was built. There are two interfaces to splicing.

. Explicit Splicing In the concretizer config, you can specify a target spec and a replacement by hash.

concretizer:
  splice:
    explicit:
    - target: mpi
      replacement: mpich/abcdef

Here, every installation that would normally use the target spec will instead use its replacement. Above, any spec using any mpi will be spliced to depend on the specific mpich installation requested. This can go wrong if you try to replace something built with, e.g., openmpi with mpich, and it is on the user to ensure ABI compatibility between target and replacement specs. This currently requires some expertise to use, but it will allow users to reuse the binaries they create across more machines and environments. . Automatic Splicing (experimental) In the concretizer config, enable automatic splicing:

  concretizer:
    splice:
      automatic: true

or run:

spack config add concretizer:splice:automatic:true

The concretizer will select splices for ABI compatibility to maximize package reuse. Packages can denote ABI compatibility using the can_splice directive. No packages in Spack yet use this directive, so if you want to use this feature you will need to add can_splice annotations to your packages. We are working on ways to add more ABI compatibility information to the Spack package repository, and this directive may change in the future.

Further documentation: https://spack.readthedocs.io/en/latest/build_settings.html#splicing https://spack.readthedocs.io/en/latest/packaging_guide.html#specifying-abi-compatibility

Broader variant propagation You can specify propagated variants like hdf5 build_type==RelWithDebInfo or trilinos ++openmp to propagate a variant to all dependencies for which it is relevant. This is valid even if the variant does not exist on the package or its dependencies. See https://spack.readthedocs.io/en/latest/basic_usage.html#variants.

Query specs by namespace Allow a package’s namespace (indicating the repository it came from) to be treated like a variant. You can request packages from particular repos like this:

spack find zlib namespace=builtin
spack find zlib namespace=myrepo

Previously, the spec syntax only allowed namespaces to be prefixes of spec names, e.g. builtin.zlib. The previous syntax still works.

spack spec respects environment settings and unify:true spack spec did not previously respect environment lockfiles or unification settings, which made it difficult to see exactly how a spec would concretize within an environment. Now it does, so the output you get with spack spec will be the same as what your environment will concretize to when you run spack concretize. Similarly, if you provide multiple specs on the command line with spack spec, it will concretize them together if unify:true is set.

Less noisy spack spec output spack spec previously showed output like this:

> spack spec /v5fn6xo
Input spec
--------------------------------
 -   /v5fn6xo

 Concretized
--------------------------------
[+]  openssl@3.3.1%apple-clang@16.0.0~docs+shared arch=darwin-sequoia-m1
...

But the input spec is redundant, and we know we run spack spec to concretize the input spec. spack spec now only shows the concretized spec.

Better output for spack find -c In an environment, spack find -c lets you search the concretized, but not yet installed, specs, just as you would the installed ones. As with spack spec, this should make it easier for you to see what will be built before building and installing it.

spack -C <env>: use an environment’s configuration without activation Spack environments allow you to associate:

You can now also pass an environment to spack -C to use only the environment’s configuration, but not the specs or lockfile.

The new spack env track command takes a non-managed Spack environment and adds a symlink to Spack’s $environments_root directory, so that it will be included for reference counting for commands like spack uninstall and spack gc. If you use free-standing directory environments, this is useful for preventing Spack from removing things required by your environments. You can undo this tracking with the spack env untrack command.

Add -t short option for spack --backtrace spack -d / --debug enables backtraces on error, but it can be very verbose, and sometimes you just want the backtrace. spack -t / --backtrace provides that option.

gc: restrict to specific specs If you only want to garbage-collect specific packages, you can now provide them on the command line. This gives users finer-grained control over what is uninstalled.

oci buildcaches now support --only=package. You can now push just a package and not its dependencies to an OCI registry. This allows dependents of non-redistributable specs to be stored in OCI registries without an error.

Externals no longer override the preferred provider. External definitions could interfere with package preferences. Now, if openmpi is the preferred mpi, and an external mpich is defined, a new openmpi will be built if building it is possible. Previously we would prefer mpich despite the preference.

Composable cflags. This release fixes a longstanding bug that concretization would fail if there were different cflags specified in packages.yaml, compilers.yaml, or on the CLI. Flags and their ordering are now tracked in the concretizer and flags from multiple sources will be merged.

Fix concretizer Unification for included environments.

The old concretizer has been removed from Spack, along with the config:concretizer config option. Spack will emit a warning if the option is present in user configuration, since it now has no effect. Spack now uses a simpler bootstrapping mechanism, where a JSON prototype is tweaked slightly to get an initial concrete spec to download.

Best-effort expansion of spec matrices has been removed. This feature did not work with the “new” ASP-based concretizer, and did not work with unify: True or unify: when_possible. Use the exclude key for the environment to exclude invalid components, or use multiple spec matrices to combine the list of specs for which the constraint is valid and the list of specs for which it is not.

The old Cray platform (based on Cray PE modules) has been removed, and platform=cray is no longer supported. Since v0.19, Spack has handled Cray machines like Linux clusters with extra packages, and we have encouraged using this option to support Cray. The new approach allows us to correctly handle Cray machines with non-SLES operating systems, and it is much more reliable than making assumptions about Cray modules. See the v0.19 release notes for more details.

The config:install_missing_compilers config option has been deprecated, and it is a no-op when set in v0.23. Our new compiler dependency model will replace it with a much more reliable and robust mechanism in v1.0.

Config options that deprecated in v0.21 have been removed in v0.23. You can now only specify preferences for compilers, targets, and providers globally via the packages:all: section. Similarly, you can only specify versions: locally for a specific package.

Spack’s old test interface has been removed, having been deprecated in v0.22.0. All builtin packages have been updated to use the new interface. See the stand-alone test documentation

The spack versions --safe-only option, deprecated since v0.21.0, has been removed.

The --dependencies and --optimize arguments to spack ci have been deprecated.

Public binary caches now include an ML stack for Linux/aarch64. We now build an ML stack for Linux/aarch64 for all pull requests and on develop. The ML stack includes both CPU-only and CUDA builds for Horovod, Hugging Face, JAX, Keras, PyTorch, scikit-learn, TensorBoard, and TensorFlow, and related packages. The CPU-only stack also includes XGBoost. See https://cache.spack.io/tag/develop/?stack=ml-linux-aarch64-cuda.

archspec has been updated to v0.2.5, with support for zen5.

Spack’s CUDA package now supports the Grace Hopper 9.0a compute capability.

Bugfix: spack find -x in environments.

Spec splices are now robust to duplicate nodes with the same name in a spec.

Cache per-compiler libc calculations for performance.

Fixed a bug in external detection for openmpi.

Mirror configuration allows username/password as environment variables.

Default library search caps maximum depth/

Unify interface for spack spec and spack solve commands.

Spack no longer RPATHs directories in the default library search path.

Improved performance of Spack database.

Enable package reuse for packages with versions from git refs.

Improved tracking of task queueing/requeueing in the installer.

Forward compatibility with Spack 0.23 packages with language dependencies.

Forward compatibility with urllib from Python 3.12.6+.

Bump vendored archspec for better aarch64 support.

Fix regression in \\{variants.X} and \\{variants.X.value} format strings.

Ensure shell escaping of environment variable values in load and activate commands.

Fix an issue where spec[pkg] considers specs outside the current DAG.

Do not halt concretization on unknown variants in externals.

Improve validation of develop config section/

Explicitly disable ccache if turned off in config, to avoid cache pollution.

Improve backwards compatibility in include_concrete.

Fix issue where package tags were sometimes repeated.

Make setup-env.sh “sourced only” by dropping execution bits.

Make certain source/binary fetch errors recoverable instead of a hard error.

Remove debug statements in package hash computation.

Remove redundant clingo warnings.

Remove hard-coded layout version.

Do not initialize previous store state in use_store.

chapel major update/

Fix reuse of externals on Linux.

Ensure parent gcc-runtime version >= child.

Ensure the latest gcc-runtime is rpath’ed when multiple exist among link deps.

Improve version detection of glibc.

Improve heuristics for solver.

Make strong preferences override reuse.

Reduce verbosity when C compiler is missing.

Make missing ccache executable an error when required.

Make every environment view containing python a venv.

Fix external detection for compilers with os but no target.

Fix version optimization for roots.

Handle common implementations of pagination of tags in OCI build caches.

Apply fetched patches to develop specs.

Avoid Windows wrappers for filesystem utilities on non-Windows.

Fix formatting issue in spack audit.

Require libiconv for iconv. Notice that glibc/musl also provide iconv, but are not guaranteed to be complete. Set packages:iconv:require:[glibc] to restore the old behavior.

protobuf: fix 3.4:3.21 patch checksum.

protobuf: update hash for patch needed when=“@3.4:3.21”.

git: bump v2.39 to 2.45; deprecate unsafe versions.

gcc: use -rpath \{rpath_dir} not -rpath=\\{rpath dir}.

Remove mesa18 and libosmesa.

Enforce consistency of gl providers.

py-matplotlib: qualify when to do a post install.

rust: fix v1.78.0 instructions.

suite-sparse: improve setting of the libs property.

netlib-lapack: provide blas and lapack together.

Add report AccountUtilizationByQOS to sreport.

AccountUtilizationByUser is able to be filtered by QOS.

Add autodetected gpus to the output of slurmd -C

Add ability to submit jobs with multiple QOS. These are sorted by priority highest being the first.

Removed the instant on feature from switch/hpe_slingshot.

slurmctld - Changed incoming RPC handling to dedicated thread pool with asynchronous handling of I/O that can be configured via conmgr_* entries under SlurmctldParameters in slurm.conf.

Add SchedulerParameters=bf_allow_magnetic_slot option. It allows jobs in magnetic reservations to be planned by backfill scheduler.

Add TopologyParam=TopoMaxSizeUnroll=# to allow --nodes=<min>-<max> for topology/block.

Add DataParserParameters slurm.conf parameter to allow setting default value for CLI --json and --yaml arguments.

Hardware collectives in switch/hpe_slingshot now requires enable_stepmgr.

Added connection related parameters to slurm.conf under SlurmctldParameters:
conmgr_max_connections: Defaults to 150 connections.

conmgr_threads: Defaults to 64 threads for slurmctld.

conmgr_use_poll: Defaults is to use epoll in Linux.

conmgr_connect_timeout: Defaults to MessageTimeout.

conmgr_read_timeout: Defaults to MessageTimeout.

conmgr_wait_write_delay: Defaults to MessageTimeout.

conmgr_write_timeout: Defaults to MessageTimeout.

Add SlurmctldParamters=ignore_constraint_validation to ignore constraint/feature validation at submission.

Add SchedulerParameters=bf_topopt_enable option to enable experimental hook to control backfill.

Remove srun --cpu-bind=rank.

Add "%b" as a file name pattern for the array task id modulo 10.

sacct - Respect --noheader for --batch-script and --env-vars.

Add sacctmgr ping command to query status of slurmdbd.

sbcast - Add --nodelist option to specify where files are transmitted to

sbcast - Add --no-allocation option to transmit files to nodes outside of a job allocation.

slurmdbd - Add -u option. This is used to determine if restarting the DBD will result in database conversion.

Remove salloc --get-user-env.

scontrol - Add --json/--yaml support to listpids.

scontrol - Add liststeps.

scontrol - Add listjobs.

scontrol show topo - Show aggregated block sizes when using topology/block.

Remove burst_buffer/lua call slurm.job_info_to_string().

job_submit/lua - Add assoc_qos attribute to job_desc to display all potential QOS’s for a job’s association.

job_submit/lua - Add slurm.get_qos_priority() function to retrieve the given QOS’s priority.

Removed fields deprecated in the Slurm-23.11 release from v0.0.42 endpoints.

Removed v0.0.39 plugins.

Set data_parser/v0.0.42+prefer_refs flag to default.

Add data_parser/v0.0.42+minimize_refs flag to inline single referenced schemas in the OpenAPI schema to get default behavior of data_parser/v0.0.41.

Rename v0.0.42 JOB_INFO field minimum_switches to required_switches to reflect the actual behavior.

Rename v0.0.42 ACCOUNT_CONDITION field assocation to association (typo).

Tag slurmdb/v0.0.42/jobs pid field deprecated.

With client commands MIN_MEMORY will show mem_per_tres if specified.

Fix errno message about bad constraint.

slurmctld - Fix crash and possible split brain issue if the backup controller handles an scontrol reconfigure while in control before the primary resumes operation.

Fix stepmgr not getting dynamic node addrs from the controller.

stepmgr - avoid Unexpected missing socket errors.

Fix scontrol show steps with dynamic stepmgr.

Deny jobs using the R: option of --signal if PreemptMode=OFF globally.

Force jobs using the R: option of --signal to be preemptable by requeue or cancel only. If PreemptMode on the partition or QOS is off or suspend, the job will default to using PreemptMode=cancel.

If --mem-per-cpu exceeds MaxMemPerCPU, the number of CPUs per task will always be increased even if --cpus-per-task was specified. This is needed to ensure each task gets the expected amount of memory.

Fix compilation issue on openSUSE Leap 15.

Fix jobs using more nodes than needed when not using -N.

Fix issue with allocation being allocated less resources than needed when using --gres-flags=enforce-binding.

select/cons_tres - Fix errors with MaxCpusPerSocket partition limit. Used CPUs/cores weren’t counted properly, nor limiting free ones to avail, when the socket was partially allocated, or the job request went beyond this limit.

Fix issue when jobs were preempted for licenses even if there were enough licenses available.

Fix srun ntasks calculation inside an allocation when nodes are requested using a min-max range.

Print correct number of digits for TmpDisk in sdiag.

Fix a regression in 24.11 which caused file transfers to a job with sbcast to not join the job container namespace.

data_parser/v0.0.40 - Prevent a segfault in the slurmrestd when dumping data with v0.0.40+complex data parser.

Remove logic to force lowercase GRES names.

data_parser/v0.0.42 - Prevent the association id from always being dumped as NULL when parsing in complex mode. Instead it will now dump the id. This affects the following endpoints:

slurmctld - Add new RPC rate limiting feature. This is enabled through SlurmctldParameters=rl_enable, otherwise disabled by default.

Make scontrol reconfigure and sending a SIGHUP to the slurmctld behave the same. If you were using SIGHUP as a 'lighter' scontrol reconfigure to rotate logs please update your scripts to use SIGUSR2 instead.

Change cloud nodes to show by default. PrivateData=cloud is no longer needed.

sreport - Count planned (FKA reserved) time for jobs running in IGNORE_JOBS reservations. Previously was lumped into IDLE time.

job_container/tmpfs - Support running with an arbitrary list of private mount points (/tmp and /dev/shm are the default, but not required).

job_container/tmpfs - Set more environment variables in InitScript.

Make all cgroup directories created by Slurm owned by root. This was the behavior in cgroup/v2 but not in cgroup/v1 where by default the step directories ownership were set to the user and group of the job.

accounting_storage/mysql - change purge/archive to calculate record ages based on end time, rather than start or submission times.

job_submit/lua - add support for log_user() from slurm_job_modify().

Run the following scripts in slurmscriptd instead of slurmctld: ResumeProgram, ResumeFailProgram, SuspendProgram, ResvProlog, ResvEpilog, and RebootProgram (only with SlurmctldParameters=reboot_from_controller).

Only permit changing log levels with srun --slurmd-debug by root or SlurmUser.

slurmctld will fatal() when reconfiguring the job_submit plugin fails.

Add PowerDownOnIdle partition option to power down nodes after nodes become idle.

Add “[jobid.stepid]” prefix from slurmstepd and “slurmscriptd” prefix from slurmcriptd to Syslog logging. Previously was only happening when logging to a file.

Add purge and archive functionality for job environment and job batch script records.

Extend support for Include files to all "configless" client commands.

Make node weight usable for powered down and rebooting nodes.

Add “Extra” field to job to store extra information other than a comment.

Add usage gathering for AMD (requires ROCM 5.5+) and NVIDIA gpus.

Add job’s allocated nodes, features, oversubscribe, partition, and reservation to SLURM_RESUME_FILE output for power saving.

Automatically create directories for stdout/stderr output files. Paths may use %j and related substitution characters as well.

Add --tres-per-task to salloc/sbatch/srun.

Allow nodefeatures plugin features to work with cloud nodes. e.g. - Powered down nodes have no active changeable features.

Make slurmstepd cgroups constrained by total configured memory from slurm.conf (NodeName=<> RealMemory=#) instead of total physical memory.

node_features/helpers - add support for the OR and parentheses operators in a --constraint expression.

slurmctld will fatal() when [Prolog|Epilog]Slurmctld are defined but are not executable.

Validate node registered active features are a super set of node’s currently active changeable features.

On clusters without any PrologFlags options, batch jobs with failed prologs nolonger generate an output file.

Add SLURM_JOB_START_TIME and SLURM_JOB_END_TIME environment variables.

Add SuspendExcStates option to slurm.conf to avoid suspending/powering down specific node states.

Add support for DCMI power readings in IPMI plugin.

slurmrestd served /slurm/v0.0.39 and /slurmdb/v0.0.39 endpoints had major changes from prior versions. Almost all schemas have been renamed and modified. Sites using OpenAPI Generator clients are highly suggested to upgrade to to using atleast version 6.x due to limitations with prior versions.

Allow for --nodelist to contain more nodes than required by --nodes.

Rename “nodes” to “nodes_resume” in SLURM_RESUME_FILE job output.

Rename “all_nodes” to “all_nodes_resume” in SLURM_RESUME_FILE output.

Add jobcomp/kafka plugin.

Add new PreemptParameters=reclaim_licenses option which will allow higher priority jobs to preempt jobs to free up used licenses. (This is only enabled for with PreemptModes of CANCEL and REQUEUE, as Slurm cannot guarantee suspended jobs will release licenses correctly.)

hpe/slingshot - add support for the instant-on feature.

Add ability to update SuspendExc* parameters with scontrol.

Add ability to restore SuspendExc* parameters on restart with slurmctld -R option.

Add ability to clear a GRES specification by setting it to "0" via “scontrol update job”.

Add SLURM_JOB_OVERSUBSCRIBE environment variable for Epilog, Prolog, EpilogSlurmctld, PrologSlurmctld, and mail ouput.

System node down reasons are appended to existing reasons, separated by ':'.

New command scrun has been added. scrun acts as an Open Container Initiative (OCI) runtime proxy to run containers seamlessly via Slurm.

Fixed GpuFreqDef option. When set in slurm.conf, it will be used if --gpu-freq was not explicitly set by the job step.

job_container.conf - Added “Dirs” option to list desired private mount points.

node_features plugins - invalid users specified for AllowUserBoot will now result in fatal() rather than just an error. MinKmemSpace.

Allow jobs to queue even if the user is not in AllowGroups when EnforcePartLimits=no is set. This ensures consistency for all the Partition access controls, and matches the documented behavior for EnforcePartLimits.

Add InfluxDBTimeout parameter to acct_gather.conf.

job_container/tmpfs - add support for expanding %h and %n in BasePath.

slurm.conf - Removed SlurmctldPlugstack option.

Add new SlurmctldParameters=validate_nodeaddr_threads=<number> option to allow concurrent hostname resolution at slurmctld startup.

Add new AccountingStoreFlags=job_extra option to store a job’s extra field in the database.

Add new “defer_batch” option to SchedulerParameters to only defer scheduling for batch jobs.

Add new DebugFlags option “JobComp” to replace “Elasticsearch”.

Add configurable job requeue limit parameter - MaxBatchRequeue - in slurm.conf to permit changes from the old hard-coded value of 5.

helpers.conf - Allow specification of node specific features.

helpers.conf - Allow many features to one helper script.

job_container/tmpfs - Add “Shared” option to support shared namespaces. This allows autofs to work with the job_container/tmpfs plugin when enabled.

acct_gather.conf - Added EnergyIPMIPowerSensors=Node=DCMI and Node=DCMI_ENHANCED.

Add new “getnameinfo_cache_timeout=<number>” option to CommunicationParameters to adjust or disable caching the results of getnameinfo().

Add new PrologFlags=ForceRequeueOnFail option to automatically requeue batch jobs on Prolog failures regardless of the job --requeue setting.

Add HealthCheckNodeState=NONDRAINED_IDLE option.

Add “explicit” to Flags in gres.conf. This makes it so the gres is not automatically added to a job’s allocation when --exclusive is used. Note that this is a per-node flag.

Moved the “preempt_” options from SchedulerParameters to PreemptParameters, and dropped the prefix from the option names. (The old options will still be parsed for backwards compatibility, but are now undocumented.)

Add LaunchParameters=ulimit_pam_adopt, which enables setting RLIMIT_RSS in adopted processes.

Update SwitchParameters=job_vni to enable/disable creating job VNIs for all jobs, or when a user requests them.

Update SwitchParameters=single_node_vni to enable/disable creating single node VNIs for all jobs, or when a user requests them.

Add ability to preserve SuspendExc* parameters on reconfig with ReconfigFlags=KeepPowerSaveSettings.

slurmdbd.conf - Add new AllResourcesAbsolute to force all new resources to be created with the Absolute flag.

topology/tree - Add new TopologyParam=SwitchAsNodeRank option to reorder nodes based on switch layout. This can be useful if the naming convention for the nodes does not natually map to the network topology.

Removed the default setting for GpuFreqDef. If unset, no attempt to change the GPU frequency will be made if --gpu-freq is not set for the step.

sacctmgr - no longer force updates to the AdminComment, Comment, or SystemComment to lower-case.

sinfo - Add -F/--future option to sinfo to display future nodes.

sacct - Rename “Reserved” field to “Planned” to match sreport and the nomenclature of the 'Planned' node.

scontrol - advanced reservation flag MAINT will no longer replace nodes, similar to STATIC_ALLOC

sbatch - add parsing for #PBS -d and #PBS -w.

scontrol show assoc_mgr will show username(uid) instead of uid in QoS section.

Add strigger --draining and -R/--resume options.

Change --oversubscribe and --exclusive to be mutually exclusive for job submission. Job submission commands will now fatal if both are set. Previously, these options would override each other, with the last one in the job submission command taking effect.

scontrol - Requested TRES and allocated TRES will now always be printed when showing jobs, instead of one TRES output that was either the requested or allocated.

srun --ntasks-per-core now applies to job and step allocations. Now, use of --ntasks-per-core=1 implies --cpu-bind=cores and --ntasks-per-core>1 implies --cpu-bind=threads.

salloc/sbatch/srun - Check and abort if ntasks-per-core > threads-per-core.

scontrol - Add ResumeAfter=<secs> option to “scontrol update nodename=”.

Add a new “nodes=” argument to scontrol setdebug to allow the debug level on the slurmd processes to be temporarily altered.

Add a new “nodes=” argument to “scontrol setdebugflags” as well.

Make it so scrontab prints client-side the job_submit() error messsage (which can be set i.e. by using the log_user() function for the lua plugin).

scontrol - Reservations will not be allowed to have STATIC_ALLOC or MAINT flags and REPLACE[_DOWN] flags simultaneously.

scontrol - Reservations will only accept one reoccurring flag when being created or updated.

scontrol - A reservation cannot be updated to be reoccurring if it is already a floating reservation.

squeue - removed unused “%s” and “SelectJobInfo” formats.

squeue - align print format for exit and derived codes with that of other components (<exit_status>:<signal_number>).

sacct - Add --array option to expand job arrays and display array tasks on separate lines.

Partial support for “--json” and “--yaml” formated outputs have been implemented for sacctmgr, sdiag, sinfo, squeue, and scontrol. The resultant data ouput will be filtered by normal command arguments. Formatting arguments will continue to be ignored.

salloc/sbatch/srun - extended the --nodes syntax to allow for a list of valid node counts to be allocated to the job. This also supports a "step count" value (e.g., --nodes=20-100:20 is equivalent to --nodes=20,40,60,80,100) which can simplify the syntax when the job needs to scale by a certain "chunk" size.

srun - add user requestible vnis with “--network=job_vni” option.

srun - add user requestible single node VNIs with the “--network=single_node_vni” option.

job_container plugins - container_p_stepd_create() function signature replaced uint32_t uid with stepd_step_rec_t* step.

gres plugins - gres_g_get_devices() function signature replaced pid_t pid with stepd_step_rec_t* step.

cgroup plugins - task_cgroup_devices_constrain() function signature removed pid_t pid.

task plugins - replace task_p_pre_set_affinity(), task_p_set_affinity(), and task_p_post_set_affinity() with task_p_pre_launch_priv() like it was back in slurm 20.11.

Allow for concurrent processing of job_submit_g_submit() and job_submit_g_modify() calls. If your plugin is not capable of concurrent operation you must add additional locking within your plugin.

Removed return value from slurm_list_append().

The List and ListIterator types have been removed in favor of list_t and list_itr_t respectively.

burst buffer plugins - add bb_g_build_het_job_script(). bb_g_get_status() - added authenticated UID and GID. bb_g_run_script() - added job_info argument.

burst_buffer.lua - Pass UID and GID to most hooks. Pass job_info (detailed job information) to many hooks. See etc/burst_buffer.lua.example for a complete list of changes. WARNING: Backwards compatibility is broken for slurm_bb_get_status: UID and GID are passed before the variadic arguments. If UID and GID are not explicitly listed as arguments to slurm_bb_get_status(), then they will be included in the variadic arguments. Backwards compatibility is maintained for all other hooks because the new arguments are passed after the existing arguments.

node_features plugins - node_features_p_reboot_weight() function removed. node_features_p_job_valid() - added parameter feature_list. node_features_p_job_xlate() - added parameters feature_list and job_node_bitmap.

New data_parser interface with v0.0.39 plugin.

The --uid option for the srun command is broken: its use may lead to the error message job <job ID> queued and waiting for resources. This does not, however, affect the sbatch command.

Removed launch plugin.

openapi/[db]v0.0.36 - plugins have been removed.

squeue - removed --array-unique option.

Deprecate AllowedKmemSpace, ConstrainKmemSpace, MaxKmemPercent, and MinKmemSpace.

openapi/[db]v0.0.37 - tagged as deprecated.