PYTHONPATHs not set for python components after releases/v0.20 #42535
Comments
|
You should be able to use: autoload: direct
hide_implicits: true # new in 0.21The point being that It looks like this is not documented well, so I'll do that next. |
|
@haampie Thank you much for this answer! I had to shift hide_implicits one level to the left otherwise spack was complaining: When I shifted it to look like this, the build started and completed, but I see the module files generated and no PYTHONPATH commands generated in the survey module file. |
It should have one for survey itself? Cause the package has
In what way exactly? Breaking / annoying / inconvenient? (There was a bug in Lmod where warnings "replacing module x with y" caused error exit codes, but should've been fixed in the meanwhile; warnings can be ignored) I guess there are two approaches:
I was hoping that (1) would have no drawbacks in general, but maybe I'm mistaken? The main upside of (1) is that you have clean and minimal module files, without duplication of environment variable values across modules, it's very composable, and you get reference counting from your module system. Current approach (2) is not implemented to set environment variables from dependencies too, but I could be convinced to do so. |
|
I've asked the person that is deploying survey on LANL systems to comment on the issues the users were having when the spack survey module loads all the dependent modules. We don't have issues with survey module loading when we use autoload: none and spack generates the PYTHONPATH for the dependencies, but that functionality disappeared after v0.20. |
Fail, how? Can you share some details about the module system type and version? Lmod with hierarchy or so?
I though the hidden modules are actually hidden. |
|
Here's the problem. When we provide Spack built packages via Spack created modulefiles with 'autoload: direct' (or anything other than autoload: none) - it works pretty well. That is, until a code team with conflicting module files actually tries to use our software. In this case, (with autoload: direct) loading Survey would cause the reload of python and other packages the code team was building themselves. This would break either survey or the code itself, depending on the order in which modulefiles were loaded. Our deployments are using Spack 0.19 currently, and things seem to work with 'autoload:none'. If this breaks in newer versions of Spack, we're going to have to patch in the old behavior. |
|
Oh, and the We also can't guarantee that the user of our survey package will have our Python loaded at all - so whatever dependencies Survey needs they must provided by the Survey modulefile itself. This all works in Spack through RPATHs for almost everything else. There are, of course, possibilities for conflicts there too - setting PYTHONPATH might override a specific module the user is providing - but it's pretty likely to work anyway. |
|
Hm okay, that sounds brittle :) So the issue comes down to:
But then how would you want to see the "conflict" resolved? Should |
|
It's incredibly fragile, but it's the best we've got at the moment.
The way I deal with this in any sort of python based software I write is to force the python either in the shebang or to wrap everything in a bash script that figures out the correct python, sets up the PYTHONPATH, and then starts the application.
…________________________________
From: Harmen Stoppels ***@***.***>
Sent: Monday, February 12, 2024 6:41 AM
To: spack/spack ***@***.***>
Cc: Ferrell, Paul Steven ***@***.***>; Comment ***@***.***>
Subject: [EXTERNAL] Re: [spack/spack] PYTHONPATHs not set for python components after releases/v0.20 (Issue #42535)
Hm okay, that sounds brittle :)
So the issue comes down to:
1. There's a Spack and user python module.
2. The module system only allows one to be loaded concurrently
3. Loading survey swaps in Spack's python module and warns about unloading theirs (or does it error?)
But then how would you want to see the "conflict" resolved?
Should survey embed prepend_path PATH <path to spack's python>/bin? I guess it has to? But if that's all squashed into the top-level package, it's gonna invalidate the user's python module too, just without warning?
—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/spack/spack/issues/42535*issuecomment-1938701253__;Iw!!Bt8fGhp8LhKGRg!BzojhI94_wnfJRwuFoklEr7GdKRJtLZQXJblvqagdQTFq-Lui6gaYefp5EBpL6z0qvgUMwFpsoa-s_KpT_Tgyhi0EQ$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AMK6D5CUWKE4A5LEBAGLTNTYTILW3AVCNFSM6AAAAABC4USOS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZYG4YDCMRVGM__;!!Bt8fGhp8LhKGRg!BzojhI94_wnfJRwuFoklEr7GdKRJtLZQXJblvqagdQTFq-Lui6gaYefp5EBpL6z0qvgUMwFpsoa-s_KpT_RaLMrBRQ$>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
Yeah, you can fix it with executable wrappers that set environment variables -- that's what nix does. That's robust, but not very flexible, since users can't override those variables then. I'm not sure how to make progress here. If you want a correctly working application, you need variables from the package and its dependencies. Autoloading makes that work, but if you don't want autoloading, squashing all variables of all dependencies into the top-level package's module sounds like the only way to guarantee that things work. However, squashing variables into a module file also has downside, cause what if you have say or is that gonna break the currently loaded I checked, and basically But for
Also, it's somewhat unclear how to deal with module hierarchies. Still feels like autoloading modules for dependencies is the most reliable way to work, even if the result is mildly annoying warnings about clashing modules. |
|
It's not just mildly annoying, it breaks one application or the other.
We for the most part don't have modulefiles for individual python packages, so that's not a problem in our case.
…________________________________
From: Harmen Stoppels ***@***.***>
Sent: Wednesday, February 14, 2024 5:44 AM
To: spack/spack ***@***.***>
Cc: Ferrell, Paul Steven ***@***.***>; Comment ***@***.***>
Subject: [EXTERNAL] Re: [spack/spack] PYTHONPATHs not set for python components after releases/v0.20 (Issue #42535)
Yeah, you can fix it with executable wrappers that set environment variables -- that's what nix does. That's robust, but not very flexible, since users can't override those variables then.
I'm not sure how to make progress here. If you want a correctly working application, you need variables from the package and its dependencies. Autoloading makes that work, but if you don't want autoloading, squashing all variables of all dependencies into the top-level package's module sounds like the only way to guarantee that things work.
However, squashing variables into a module file also has downside, cause what if you have say
module load python
module load py-pkg # depends on python, so py-pkg has a superset of all variables that the python package has
module unload py-pkg
or
module load mpich
module load pkg-that-uses-mpich # same, contains whatever is in mpich
module unload pkg-that-uses-mpich
is that gonna break the currently loaded python / mpich package because common/squashed variables are unset after unload?
I checked, and basically prepend-path commands are OK in both lmod and tcl modules. If two modules prepend the same path, it shows up only once in PATH, and the path only gets removed after unloading both.
But for setenv this is not the case. If two modules setenv FOO bar, then loading both and unloading just one of the two unsets the variable, breaking the other loaded module.
environment-modules has https://modules.readthedocs.io/en/latest/modulefile.html#mfcmd-pushenv<https://urldefense.com/v3/__https://modules.readthedocs.io/en/latest/modulefile.html*mfcmd-pushenv__;Iw!!Bt8fGhp8LhKGRg!G5Co4A7NWPaVWPhsuD_qqphQ3-QITn-eJvODLZHAU9nu3LsWumAH4O9-hFEnkWkTOc9s411FxLbdtQBdkw4aC6-WRw$> but it's only supported from version 5.1.
Also, it's somewhat unclear how to deal with module hierarchies.
Still feels like autoloading modules for dependencies is the most reliable way to work, even if the result is mildly annoying warnings about clashing modules.
—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/spack/spack/issues/42535*issuecomment-1943698485__;Iw!!Bt8fGhp8LhKGRg!G5Co4A7NWPaVWPhsuD_qqphQ3-QITn-eJvODLZHAU9nu3LsWumAH4O9-hFEnkWkTOc9s411FxLbdtQBdkw5HgrKGcQ$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AMK6D5F2YIX3OS4OIP2EGTLYTSWRDAVCNFSM6AAAAABC4USOS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBTGY4TQNBYGU__;!!Bt8fGhp8LhKGRg!G5Co4A7NWPaVWPhsuD_qqphQ3-QITn-eJvODLZHAU9nu3LsWumAH4O9-hFEnkWkTOc9s411FxLbdtQBdkw48DtcN-w$>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
I still don't understand what you mean by "breaks". |
|
I just tested survey build using spack 0.20.0 and 0.22.0
module load openmpi/4.1.1-gcc_10.3.0 survey/1.0.9-openmpi.4 module list
now 0.22.0: module load openmpi/4.1.1-gcc_10.3.0 survey/1.0.9.0126-gcc-10.3.0-27eia6d module list
|
|
OK, I thought we were hiding hidden modules also in |
|
Trying to come up with a solution that is mostly correct and has good UX:
Then (1) and (2) ensure that And (3) is just for user experience, assuming that you find all the |
The user application provides its own Spack built modules that conflict with those provided by the Spack built programming environment. Loading survey forces the reload of those conflicting modules when I'd love to see a per-package 'view' of the python dependencies - basically a virtual env that each python application runs under. You would only have to change the path to python for the application, and python would know about its paths from their just like venv setups do. |
|
To offer an alternative approach, I ran into this same issue with Survey and overcame it using Spack views. The idea is that we create a view to collect the Python modules into a single site-packages directory which can then be set in the modules configuration so that we are adding 1 path to modules.yaml: spack.yaml: |
|
for our use-case, we would prefer that Spack adds the prepend path to PYTHONPATH statements for each python module in the survey module without creating new modules, hidden or otherwise |
|
We were thinking of doing something similar. We'll have to try that out.
…________________________________
From: garylawson ***@***.***>
Sent: Wednesday, February 28, 2024 11:06 AM
To: spack/spack ***@***.***>
Cc: Ferrell, Paul Steven ***@***.***>; Comment ***@***.***>
Subject: [EXTERNAL] Re: [spack/spack] PYTHONPATHs not set for python components after releases/v0.20 (Issue #42535)
for our use-case, we would prefer that Spack adds the prepend path to PYTHONPATH statements for each python module in the survey module without creating new modules, hidden or otherwise
—
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/spack/spack/issues/42535*issuecomment-1969554588__;Iw!!Bt8fGhp8LhKGRg!GV67H1cOCeiYq4T2xHeVulglXSV1p5DK9l2cDQS9uh5AgCQ9yY900txdS5BbAtatHeQ_2gYRw-1khX6XV7M3kwIdYQ$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AMK6D5HYB2VDXZRBA66BE4DYV5W3DAVCNFSM6AAAAABC4USOS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRZGU2TINJYHA__;!!Bt8fGhp8LhKGRg!GV67H1cOCeiYq4T2xHeVulglXSV1p5DK9l2cDQS9uh5AgCQ9yY900txdS5BbAtatHeQ_2gYRw-1khX6XV7P1PbPIyg$>.
You are receiving this because you commented.Message ID: ***@***.***>
|
The following was sufficient to implement (1) and (2) on one of our clusters: It's been deployed for a week and works quite well. A release implementation should probably make this a parameter configurable in |
|
Sorry, missed a lot of messages here. @camillescottatwork see #42743, which does exactly that. If it works for you please comment there. |
|
@haampie awesome! I'm actually deploying another spack instance currently, so I'll let you know how it works for us. |
Discussed in #42433
Originally posted by jgalarowicz February 1, 2024
I tried to use spack releases instead of the development branch to build survey in hopes of figuring out why we don't get the PYTHONPATH statements in the survey module file.
I'm asking spack not to load the module files for all the components by using autoload: none feature.
When building survey with spack releases/v0.20, spack puts all the PYTHONPATH statements into the module file for survey. Releases after v0.20 do not put the PYTHONPATH statements in.
I would like to know how we can get this functionality back.
With releases/v0.21 and above we only get the PYTHONPATH for survey, but not the components.
When we try to execute survey we get python module not found error messages.
This is from releases/v0.20 generated spack survey module file:
Thanks,
Jim G
The text was updated successfully, but these errors were encountered: