Release v0.22.0 (2024-05-12) · spack/spack

v0.22.0 is a major feature release.

Features in this release

Compiler dependencies

We are in the process of making compilers proper dependencies in Spack, and a number
of changes in v0.22 support that effort. You may notice nodes in your dependency
graphs for compiler runtime libraries like gcc-runtime or libgfortran, and you
may notice that Spack graphs now include libc. We've also begun moving compiler
configuration from compilers.yaml to packages.yaml to make it consistent with
other externals. We are trying to do this with the least disruption possible, so
your existing compilers.yaml files should still work. We expect to be done with
this transition by the v0.23 release in November.
- #41104: Packages compiled with %gcc on Linux, macOS and FreeBSD now depend on a
  new package gcc-runtime, which contains a copy of the shared compiler runtime
  libraries. This enables gcc runtime libraries to be installed and relocated when
  using a build cache. When building minimal Spack-generated container images it is
  no longer necessary to install libgfortran, libgomp etc. using the system package
  manager.
- #42062: Packages compiled with %oneapi now depend on a new package
  intel-oneapi-runtime. This is similar to gcc-runtime, and the runtimes can
  provide virtuals and compilers can inject dependencies on virtuals into compiled
  packages. This allows us to model library soname compatibility and allows
  compilers like %oneapi to provide virtuals like sycl (which can also be
  provided by standalone libraries). Note that until we have an agreement in place
  with intel, Intel packages are marked redistribute(source=False, binary=False)
  and must be downloaded outside of Spack.
- #43272: changes to the optimization criteria of the solver improve the hit-rate of
  buildcaches by a fair amount. The solver more relaxed compatibility rules and will
  not try to strictly match compilers or targets of reused specs. Users can still
  enforce the previous strict behavior with require: sections in packages.yaml.
  Note that to enforce correct linking, Spack will not reuse old %gcc and
  %oneapi specs that do not have the runtime libraries as a dependency.
- #43539: Spack will reuse specs built with compilers that are not explicitly
  configured in compilers.yaml. Because we can now keep runtime libraries in build
  cache, we do not require you to also have a local configured compiler to use the
  runtime libraries. This improves reuse in buildcaches and avoids conflicts with OS
  updates that happen underneath Spack.
- #43190: binary compatibility on linux is now based on the libc version,
  instead of on the os tag. Spack builds now detect the host libc (glibc or
  musl) and add it as an implicit external node in the dependency graph. Binaries
  with a libc with the same name and a version less than or equal to that of the
  detected libc can be reused. This is only on linux, not macos or Windows.
- #43464: each package that can provide a compiler is now detectable using spack external find. External packages defining compiler paths are effectively used as
  compilers, and spack external find -t compiler can be used as a substitute for
  spack compiler find. More details on this transition are in
  the docs
Improved spack find UI for Environments

If you're working in an enviroment, you likely care about:
- What are the roots
- Which ones are installed / not installed
- What's been added that still needs to be concretized
We've tweaked spack find in environments to show this information much more
clearly. Installation status is shown next to each root, so you can see what is
installed. Roots are also shown in bold in the list of installed packages. There is
also a new option for spack find -r / --only-roots that will only show env
roots, if you don't want to look at all the installed specs.

More details in #42334.
Improved command-line string quoting

We are making some breaking changes to how Spack parses specs on the CLI in order to
respect shell quoting instead of trying to fight it. If you (sadly) had to write
something like this on the command line:
```
spack install zlib cflags=\"-O2 -g\"
```
That will now result in an error, but you can now write what you probably expected
to work in the first place:
```
spack install zlib cflags="-O2 -g"
```
Quoted can also now include special characters, so you can supply flags like:
```
spack intall zlib ldflags='-Wl,-rpath=$ORIGIN/_libs'
```
To reduce ambiguity in parsing, we now require that you not put spaces around =
and == when for flags or variants. This would not have broken before but will now
result in an error:
```
spack install zlib cflags = "-O2 -g"
```
More details and discussion in #30634.
Revert default spack install behavior to --reuse

We changed the default concretizer behavior from --reuse to --reuse-deps in
#30990 (in v0.20), which meant that every spack install invocation would
attempt to build a new version of the requested package / any environment roots.
While this is a common ask for upgrading and for developer workflows, we don't
think it should be the default for a package manager.

We are going to try to stick to this policy:
1. Prioritize reuse and build as little as possible by default.
2. Only upgrade or install duplicates if they are explicitly asked for, or if there
  is a known security issue that necessitates an upgrade.
With the install command you now have three options:
- --reuse (default): reuse as many existing installations as possible.
- --reuse-deps / --fresh-roots: upgrade (freshen) roots but reuse dependencies if possible.
- --fresh: install fresh versions of requested packages (roots) and their dependencies.
We've also introduced --fresh-roots as an alias for --reuse-deps to make it more clear
that it may give you fresh versions. More details in #41302 and #43988.
More control over reused specs

You can now control which packages to reuse and how. There is a new
concretizer:reuse config option, which accepts the following properties:
- roots: true to reuse roots, false to reuse just dependencies
- exclude: list of constraints used to select which specs not to reuse
- include: list of constraints used to select which specs to reuse
- from: list of sources for reused specs (some combination of local,
  buildcache, or external)
For example, to reuse only specs compiled with GCC, you could write:
```
concretizer:
   reuse:
     roots: true
     include:
     - "%gcc"
```
Or, if openmpi must be used from externals, and it must be the only external used:
```
concretizer:
  reuse:
    roots: true
    from:
    - type: local
      exclude: ["openmpi"]
    - type: buildcache
      exclude: ["openmpi"]
    - type: external
      include: ["openmpi"]
```

New redistribute() directive

Some packages can't be redistributed in source or binary form. We need an explicit
way to say that in a package.

Now there is a redistribute() directive so that package authors can write:

class MyPackage(Package):
    redistribute(source=False, binary=False)

Like other directives, this works with when=:

class MyPackage(Package):
    # 12.0 and higher are proprietary
    redistribute(source=False, binary=False, when="@12.0:")

    # can't redistribute when we depend on some proprietary dependency
    redistribute(source=False, binary=False, when="^proprietary-dependency")

New commands, options, and directives

Allow packages to be pushed to build cache after install from source (#42423)
spack develop: stage build artifacts in same root as non-dev builds #41373
- Don't delete spack develop build artifacts after install (#43424)
spack find: add options for local/upstream only (#42999)
spack logs: print log files for packages (either partially built or installed) (#42202)
patch: support reversing patches (#43040)
develop: Add -b/--build-directory option to set build_directory package attribute (#39606)
spack list: add --namesapce / --repo option (#41948)
directives: add checked_by field to license(), add some license checks
spack gc: add options for environments and build dependencies (#41731)
Add --create to spack env activate (#40896)

Performance improvements

environment.py: fix excessive re-reads (#43746)
ruamel yaml: fix quadratic complexity bug (#43745)
Refactor to improve spec format speed (#43712)
Do not acquire a write lock on the env post install if no views (#43505)
asp.py: fewer calls to spec.copy() (#43715)
spec.py: early return in __str__
avoid jinja2 import at startup unless needed (#43237)

Other new features of note

archspec: update to v0.2.4: support for Windows, bugfixes for neoverse-v1 and
neoverse-v2 detection.
spack config get/blame: with no args, show entire config
spack env create <env>: dir if dir-like (#44024)
ASP-based solver: update os compatibility for macOS (#43862)
Add handling of custom ssl certs in urllib ops (#42953)
Add ability to rename environments (#43296)
Add config option and compiler support to reuse across OS's (#42693)
Support for prereleases (#43140)
Only reuse externals when configured (#41707)
Environments: Add support for including views (#42250)

Binary caches

Build cache: make signed/unsigned a mirror property (#41507)
tools stack

Removals, deprecations, and syntax changes

remove dpcpp compiler and package (#43418)
spack load: remove --only argument (#42120)

Notable Bugfixes

repo.py: drop deleted packages from provider cache (#43779)
Allow + in module file names (#41999)
cmd/python: use runpy to allow multiprocessing in scripts (#41789)
Show extension commands with spack -h (#41726)
Support environment variable expansion inside module projections (#42917)
Alert user to failed concretizations (#42655)
shell: fix zsh color formatting for PS1 in environments (#39497)
spack mirror create --all: include patches (#41579)

Spack community stats

7,994 total packages; 525 since v0.21.0
- 178 new Python packages, 5 new R packages
358 people contributed to this release
- 344 committers to packages
- 45 committers to core

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v0.22.0 (2024-05-12)