Cannot reuse specs with externals on linux #44085
Comments
|
Todd and Harmen have been working this issue from an alternative angle, just openmpi instead of intel compiler + intel mpi: $ spack spec openmpi@4.1.6~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~internal-libevent~internal-pmix~java~legacylaunchers~lustre~memchecker~openshmem~orterunprefix+pmi~romio+rsh~singularity~static+vt+wrapper-rpath schedulers=slurmCommit info from the smaller environment where we've been working: Attached index.json from the system where we are examining the openmpi specifically. Tagging the dynamic duo who've been looking into this from slack. |
|
spack spec --json sombrero ^openmpi spack spec --json gromacs@2024.1 ^openmpi |
|
spack spec --json openmpi/boemme |
|
Looking into this. It seems the issue is that we do not emit a |
alalazo
changed the title
|
I'll submit a PR asap, but this diff: diff --git a/lib/spack/spack/solver/libc_compatibility.lp b/lib/spack/spack/solver/libc_compatibility.lp
index 28c7c57fda..5efbcd8667 100644
--- a/lib/spack/spack/solver/libc_compatibility.lp
+++ b/lib/spack/spack/solver/libc_compatibility.lp
@@ -10,12 +10,13 @@
%=============================================================================
% A package cannot be reused if the libc is not compatible with it
-:- provider(node(X, LibcPackage), node(0, "libc")),
- attr("version", node(X, LibcPackage), LibcVersion),
- attr("hash", node(R, ReusedPackage), Hash),
- % Libc packages can be reused without the "compatible_libc" attribute
- ReusedPackage != LibcPackage,
- not attr("compatible_libc", node(R, ReusedPackage), LibcPackage, LibcVersion).
+error(100, "Cannot reuse {0} since we cannot determine libc compatibility", ReusedPackage)
+ :- provider(node(X, LibcPackage), node(0, "libc")),
+ attr("version", node(X, LibcPackage), LibcVersion),
+ attr("hash", node(R, ReusedPackage), Hash),
+ % Libc packages can be reused without the "compatible_libc" attribute
+ ReusedPackage != LibcPackage,
+ not attr("compatible_libc", node(R, ReusedPackage), LibcPackage, LibcVersion).
% Check whether the DAG has any built package
has_built_packages() :- build(X), not external(X).
should be sufficient to give a better error message: # spack solve sombrero ^/boemme
==> Error: concretization failed for the following reasons:
1. Cannot reuse slurm since we cannot determine libc compatibility
|
|
@snowbird294 This other diff should hopefully fix your issue: diff --git a/lib/spack/spack/solver/asp.py b/lib/spack/spack/solver/asp.py
index 0083dbc070..10b9b2b66b 100644
--- a/lib/spack/spack/solver/asp.py
+++ b/lib/spack/spack/solver/asp.py
@@ -1939,6 +1939,13 @@ def _spec_clauses(
for virtual in virtuals:
clauses.append(fn.attr("virtual_on_incoming_edges", spec.name, virtual))
+ # If the spec is external and concrete, we allow all the libcs on the system
+ if spec.external and spec.concrete and using_libc_compatibility():
+ for libc in self.libcs:
+ clauses.append(
+ fn.attr("compatible_libc", spec.name, libc.name, libc.version)
+ )
+
# add all clauses from dependencies
if transitive:
# TODO: Eventually distinguish 2 deps on the same pkg (build and link)
|
|
I super appreciate the error message update and the patch. I can confirm that the solver change worked like a charm: Abbreviated: |
|
The patch worked for openmpi and my most recent intel-oneapi-compiler+mpi, but I'm getting the bug when I move to building against some older compiler+mpis. Do i need to specify them differently for stuff that was built with previous versions of spack? $ spack spec hdf5@1.14.3 +fortran +mpi %oneapi@2022.2.1 ^intel-oneapi-mpi/zfmfvyc spack spec --json hdf5@1.14.3 +fortran +mpi %oneapi@2022.2.1 ^intel-oneapi-mpi@2021.7.1%oneapi@2022.2.1 spack spec --json intel-oneapi-mpi@2021.7.1%oneapi@2022.2.1 spack spec --json /zfmfvy spack spec hdf5@1.14.3 +fortran +mpi %oneapi@2024.1 ^intel-oneapi-mpi@2021.12.0 |
If those specs don't have any runtime, and they are not external, then this is expected. The general issue is that specs without a runtime may rely on some compiler being installed in specific places. To avoid installing something broken, we exclude those specs from reuse. |
Can you elaborate on "don't have any runtime"? I assume you mean the packages built pre-0.21.2 release are now in the funny state where they don't have a listed gcc-runtime (or similar) listed as a dependency. For compatibility, could I list the compilers/mpis as external specs to "trick" spack into thinking the spec is fine to reuse? |
Specs that don't have dependencies on any external libc (on linux) or that are compiled with The motivation for adopting these rules is that, in v0.22, we also relaxed a lot the criteria for reuse, and now Spack tries to reuse across operating systems and compilers. Since that might mean getting broken installations from old buildcaches, we decided to filter out those old specs.
Yes, but I wouldn't advise that for each and every spec. If there is the possibility, I would just rebuild specs with v0.22. |
I absolutely want to rebuild, but we're in a situation where if I do that, the hash's change, and some number of our users would have downstream compiled packages that would break. It's not a great situation. Having a messy work-around for now is very beneficial, while I encourage users to move to new versions/specs that exist post spack-v0.22 Thanks much for your help, patch, and insights. |
|
A usage question. I've defined intel-oneapi-mpi@2021.7.1 as an external, but when I go to install hdf5, it identifies the oneapi with a different hash: packages.yaml spack install hdf5@1.14.3 +fortran +mpi %oneapi@2022.2.1 ^intel-oneapi-mpi@2021.7.1 this wouldn't be a problem, if modules didn't already exist for intel-oneapi-mpi-zfmfvyc. I need that hash to be consistent so that the new hdf5 modules are installed in the correct place, without me just manually moving them. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Steps to reproduce the issue
Error message
Error message for hdf5
Error message for fftw
Information on your system
spack find intel-oneapi-mpi@2021.7.1 %oneapi@2022.2.1
cat /apps/spack-managed/oneapi-2022.2.1/intel-oneapi-mpi-2021.7.1-zfmfvyc4a4wkhgxe2f34irg663e6dcoh/.spack/spack-build-env.txt | grep SPEC
Specifying the exact spec of mpi returns a different concretized mpi:
Happens with fftw3 too
Additional information
the build didn't exactly fail, but this is most relevant to the build stage. May indicate a core spack issue, but I don't want to start with that conclusion. This is present in multiple build packages. For some reason, specifying the mpi to build against does not match any found mpi, even when pashed by hash. This occurs with openmpi and other intel-oneapi-mpi installations as well.
@brtnfld @gheber @hyoklee @lkurz @lrknox
General information
spack debug reportand reported the version of Spack/Python/Platformspack maintainers <name-of-the-package>and @mentioned any maintainersThe text was updated successfully, but these errors were encountered: