Centralized repository for HPC site configurations

[JBlaschke]

This is more of a comment than an Issue: I have started working on: https://pypi.org/project/mpi4py-installer/

My aim for that project is to have a centralized place for HPC centers to keep their mpi4py pip install configurations. The overall philosophy is:

1. automatic site detection + sensible defaults
2. the ability for advanced users to take control (e.g. using the --variant flag)
3. easy to add and maintain site detection and configuration logic (e.g. supporting module load, etc as part of the install)

I think it would be nice to have a central place to keep and maintain the site-specific logic. This would make it easier for HPC users to quickly install mpi4py without needing manage their own deployment recipes. I can see pros and cons with up-streaming this into mpi4py, or keeping is separate.

Let me know what you think.

[dalcinl]

@JBlaschke I totally understand the frustration and motivation to put this thing together. Below some comments to warm-up for further discussion.

• I don't think I would upstream this to mpi4py, simply because all this stuff will require external contribution and maintenance. Contributors come and go, but as years pass, I'm the ultimate person responsible for keeping this project alive, and I'm don't feel comfortable committing to things that are not under my control (eg. NERSC access) to take immediate action upon any requests for help from users.
• However, it is totally fine to keep this stuff in a separate repository. We could eventually host it within mpi4py's GitHub organization for increased visibility.
• I'm not totally sure about using Python as the implementation language vs let say implementing everything is a nice bash script. For example, how do you plan to support module load ... and related environment setups?
• There is a minor detail I don't like about your implementation. You devised the whole thing as a Python package that you have to pip install first, next execute with python -m .... I think we can do much better using a (zip application)[https://docs.python.org/3/library/zipapp.html], so you just download a throw-away zip file (e.g with curl or wget) to the working directory, and then you execute it. Anyway, a zip app can be provided for those who want to use it, and the installer can still be obtained via pip. After all, you already took the mpi4py_installer name form the PyPI namespace.

[JBlaschke]

Hi @dalcinl -- thank you for the feedback! Here are some thoughts.

It makes sense that upstreaming this would have implications about long-term support. I'm not married to either approach (upstream vs separate repo), as long as it is easily findable.

Re the bash-script idea: this project is born out of my own massive collection of bash scripts that was becoming unwieldy to maintain. I think python is a much better language to:

• handle site-detection logic + the subtle variants that are available at each site
• have a convenient way for advanced users to overwrite the automatic defaults
• different log levels, which we can use to diagnose issues when answering trouble tickets

All of this becomes a pain to do in bash. Lastly, this project needs the contributions of other sites to be successful, and I think we're far more likely to get those in python than bash (this is my subjective experience based on working with several teams).

I am happy to add the curl app as a separate mode of operation. I like the "pip install installer" => "use installer to install mpi4py" workflow because it ensures users haven't forgotten to activate the right virtualenv / conda environment. The throwaway zip approach might be more convenient for other users though, so I'll take a look at how much work it would be to maintain both.

[JBlaschke]

Based on your suggestion, I have now built a zipapp which can be easily invoked using:

curl -fsSL https://raw.githubusercontent.com/JBlaschke/mpi4py-installer/main/shiv/install.sh | sh

This is in addition to the pip-installable approach.

I forgot to mention earlier that this installer is compatible with module load commands -- they are specified in each site config.

[dalcinl]

Great. I have a minor security-related suggestion: instead of writing directly to /tmp, perhaps is better to tmpdir=$(mktemp -d) instead?

[dalcinl]

Another thing. I believe using GitHub Releases to store the *.pyz artifacts is a better approach. This way, you can simply use

curl -fsSL  https://github.com/JBlaschke/mpi4py-installer/releases/latest/download/mpi4py-installer.pyz

in your bash script and get the latest release. IMHO, this approach will require much less maintenance.