Getting mpi4py to ignore conda's MPI

[bernstei]

I'm trying to get mpi4py installed on a Linux HPC where the python comes from conda, but the MPI I want to use does not. Unfortunately, this leads to the need to install mpi4py with pip, which I gather from pervious issues isn't entirely well behaved when python comes from conda (but at least the compilers conda package isn't installed according to conda list). I tried it with just getting mpicc from the PATH, or specifying explicitly with MPICC, but in either case I end up with mixed up versioning for the mpi library. In particular, when I do

ldd ~/.local/lib/python3.9/site-packages/mpi4py/MPI.cpython-39-x86_64-linux-gnu.so

I get (among other lines)

	libmpi.so.12 => /app/CSE/CSE.20220624/Release/anaconda3-2022.05/lib/libmpi.so.12 (0x00007f2417676000)

instead of the path that's consistent with my mpicc, which is

/app/openmpi/platforms/openmpi-3.1.3/gcc-8.2.0/lib/....

Is there any obvious way to figure out why it's linking to the wrong mpi library? If I run anyway I get a missing symbol at runtime, which isn't particular surprising if it built with the system openmpi but is running with the conda one, or vice versa.

[bernstei]

And I checked that the desired MPI's path is ahead of conda's in LD_LIBRARY_PATH. However, when I pip install with -v I see the following link command

  /app/openmpi/platforms/openmpi-3.1.3/gcc-8.2.0/bin/mpicc -shared -Wl,-rpath,/app/CSE/CSE.20220624/Release/anaconda3-2022.05/lib -Wl,-rpath-link,/app/CSE/CSE.20220624/Release/anaconda3-2022.05/lib -L/app/CSE/CSE.20220624/Release/anaconda3-2022.05/lib -L/app/CSE/CSE.20220624/Release/anaconda3-2022.05/lib -Wl,-rpath,/app/CSE/CSE.20220624/Release/anaconda3-2022.05/lib -Wl,-rpath-link,/app/CSE/CSE.20220624/Release/anaconda3-2022.05/lib -L/app/CSE/CSE.20220624/Release/anaconda3-2022.05/lib build/temp.linux-x86_64-cpython-39/src/MPI.o -Lbuild/temp.linux-x86_64-cpython-39 -ldl -o build/lib.linux-x86_64-cpython-39/mpi4py/MPI.cpython-39-x86_64-linux-gnu.so

Could the issue be that the conda path is explicitly added by the mpicc link command as rpath?

[bernstei]

I also tried to modify the rpath with chrpath, which worked to change rpath, but unfortunately the version numbers of the one it's looking for (12) is not the same as the one that's provided by the HPC's mpi (40), so it still doesn't link to the right one.

I guess that at the .so's link time it's finding the conda MPI, rather than the mpicc's MPI, and then it hard-wires that version of the so file.

[leofang]

Sorry, I am not sure what you're trying to achieve. Let me ask this: Have you tried the "external MPI" trick that conda-forge designed to support HPC clusters?
https://conda-forge.org/docs/user/tipsandtricks.html#using-external-message-passing-interface-mpi-libraries
Basically, if you are using MPICH/Open MPI (or their ABI-compatible variants), you can have an empty MPI package with real mpi4py & co installed from conda-forge. Then, you just need to ensure your own MPI implementation is supplied and locatable by the dynamic linker. No need to rebuild anything.

[bernstei]

I'm trying to get an mpi4py that uses the centrally installed MPI instead of the one that's installed as part of conda. Sounds like that trick you linked to should do exactly what I want, if I can get it to work (and I worry because the conda environment already installed at the HPC has "real" conda mpi packages installed). However, it's not working. I get

(pymatnext) [noamb@mustang09 ~ ]$ conda install "mpich=x.y.z=external_*"
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - mpich==x.y.z[build=external_*]

Current channels:

  - https://conda.anaconda.org/conda-forge/linux-64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

Are those "x.y.z" placeholders, or are they meant to be used literally, like I tried?

[leofang]

x.y.z should not be taken literally, it should be the version you need, for example replace it with 3.* to get MPICH 3 (mpich=3.*=external_*).

[dalcinl]

@leofang Maybe we should convert this issue on a discussion under the Q&A category?

[bernstei]

Is there any way to get conda to ignore the system-wide "real" mpich installation? I activated an environment and installed the "external" mpich version, as the hint suggested, but when I pip install mpi4py it still resolves the MPI.*.so dependency on libmpi.so with the one from the central conda-installed mpich.

Or is there no way to do this with the centrally installed conda because that one already includes a real mpich, and I have to ignore it and install my own entire conda/python ecosystem (unless I can convince the sysadmins to replace the real mpich with the fake one)?

[leofang]

It sounds very weird to me, I am not sure I follow. Is your system MPI provided through conda or what?

I would think that if you create a new conda environment (conda create -n my_env -c conda-forge python ...; conda activate my_env), get your stuff installed (conda install -c conda-forge "mpich=3.*=external_*" mpi4py ...), and activate the system MPI (in some clusters they'd do, e.g., module load mpich/3.1.2), you shouldn't get packages interfering from outside of your current conda environment (my_env), which is the whole point of virtual envs. Basically, the idea is to only allow limited leaking (system MPI) into the current conda env.

[bernstei]

The MPI I want to use is not provided through conda, but the centrally installed conda does have a "real" mpich package installed. I've set aside for now my earlier attempt to install mpi4py using pip, and I tried to follow your suggestion for mpi4py via conda. However, now I'm getting a conda conflict.

I ran your conda create ... command (with nothing after the python), then the conda activate ..., and it seemed fine. However, when I tried to install the external mpich and mpi4py I got a conflict:

$ conda install -c conda-forge "mpich=3.*=external_*" mpi4py 
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: / 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed                                                                                                                     

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

.
.
.

followed by a few tens of lines of conflicts, listing many packages, and ending with

Package python_abi conflicts for:
mpi4py -> python_abi[version='2.7.*|3.10.*|3.11.*|3.8|3.9|3.9.*|3.8.*|3.7.*|3.7|3.6.*|3.6',build='*_cp36m|*_pypy39_pp73|*_pypy38_pp73|*_cp311|*_cp39|*_cp310|*_cp38|*_cp37m|*_pypy37_pp73|*_pypy36_pp73|*_cp27mu']
mpi4py -> python[version='>=3.8,<3.9.0a0'] -> python_abi[version='3.10.*|3.8.*',build='*_graalpy223_38_native|*_graalpy230_310_native']The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.17=0
  - feature:|@/linux-64::__glibc==2.17=0
  - bzip2 -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
  - libffi -> libgcc-ng[version='>=9.4.0'] -> __glibc[version='>=2.17']
  - libgcc-ng -> __glibc[version='>=2.17']
  - libnsl -> libgcc-ng[version='>=9.4.0'] -> __glibc[version='>=2.17']
  - libuuid -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
  - mpi4py -> libgcc-ng[version='>=10.3.0'] -> __glibc[version='>=2.17']
  - ncurses -> libgcc-ng[version='>=10.3.0'] -> __glibc[version='>=2.17']
  - openssl -> libgcc-ng[version='>=10.3.0'] -> __glibc[version='>=2.17']
  - python=3.11 -> libgcc-ng[version='>=11.2.0'] -> __glibc[version='>=2.17']
  - readline -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
  - tk -> libgcc-ng[version='>=9.4.0'] -> __glibc[version='>=2.17']
  - xz -> libgcc-ng[version='>=7.5.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.17

Note that strict channel priority may have removed packages required for satisfiability.

[leofang]

It'd be better to see

1. the output of conda list from your env
2. replace conda by mamba and retry with mamba install ...
   • you can get it by conda install -c conda-forge mamba
   • mamba gives much better error output when it comes to conflicts, and is much faster in resolving dependencies
   • if an error happens, please attach the full output, not the truncated one.

[bernstei]

OK, after doing conda install -c conda-forge mamba, which seemed to work, I get

(pymatnext) [noamb@mustang04 ~ ]$ conda list
# packages in environment at /p/home/noamb/.conda/envs/pymatnext:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
boltons                   23.0.0             pyhd8ed1ab_0    conda-forge
brotli-python             1.0.9           py311ha362b79_9    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.19.1               hd590300_0    conda-forge
ca-certificates           2023.5.7             hbcca054_0    conda-forge
certifi                   2023.5.7           pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py311h409f033_3    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     23.5.2          py311h38be061_0    conda-forge
conda-package-handling    2.0.2              pyh38be061_0    conda-forge
conda-package-streaming   0.8.0              pyhd8ed1ab_0    conda-forge
cryptography              41.0.2          py311h63ff55d_0    conda-forge
fmt                       9.1.0                h924138e_0    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
jsonpointer               2.0                        py_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.1               h659d440_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libarchive                3.6.2                h039dbb9_1    conda-forge
libcurl                   8.2.0                hca28451_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libmamba                  1.4.9                h658169a_0    conda-forge
libmambapy                1.4.9           py311h527f279_0    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsolv                   0.7.24               hfc55251_1    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxml2                   2.11.4               h0d562d8_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     1.4.9           py311h3072747_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
openssl                   3.1.1                hd590300_1    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pip                       23.2               pyhd8ed1ab_0    conda-forge
pluggy                    1.2.0              pyhd8ed1ab_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.4           py311hd4cff14_1    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.11.4          hab00c5b_0_cpython    conda-forge
python_abi                3.11                    3_cp311    conda-forge
readline                  8.2                  h8228510_1    conda-forge
reproc                    14.2.4               h0b41bf4_0    conda-forge
reproc-cpp                14.2.4               hcb278e6_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.17.32         py311h459d7ec_0    conda-forge
ruamel.yaml.clib          0.2.7           py311h2582759_1    conda-forge
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
tqdm                      4.65.0             pyhd8ed1ab_1    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
urllib3                   2.0.4              pyhd8ed1ab_0    conda-forge
wheel                     0.40.0             pyhd8ed1ab_1    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml-cpp                  0.7.0                h27087fc_2    conda-forge
zstandard                 0.19.0          py311haa97af0_2    conda-forge
zstd                      1.5.2                hfc55251_7    conda-forge

The install of mpi4py still fails with (I've dropped about a couple hundred copies of the same output line):

(pymatnext) [noamb@mustang04 ~ ]$ mamba install -c conda-forge "mpich=3.*=external_*" mpi4py

Looking for: ['mpich=3[build=external_*]', 'mpi4py']

pkgs/main/linux-64 (check zst)                      Checked  0.1s
pkgs/main/noarch (check zst)                       Checked  0.0s
pkgs/r/linux-64 (check zst)                        Checked  0.0s
pkgs/r/noarch (check zst)                          Checked  0.0s
[+] 0.1s
conda-forge/linux-64 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━   0.0 B /  ??.?MB @  ??.?MB/s  0.1s
pkgs/main/noarch                                   696.1kB @   5.9MB/s  0.1s
pkgs/r/linux-64                                      1.2MB @   7.7MB/s  0.2s
pkgs/r/noarch                                                 No change
pkgs/main/linux-64                                   5.2MB @  26.0MB/s  0.2s
conda-forge/noarch                                  11.7MB @  20.4MB/s  0.6s
conda-forge/linux-64                                29.2MB @  28.0MB/s  1.1s

Pinned packages:
  - python 3.11.*


warning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY
warning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY
.
.
.
warning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY
warning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY
warning  libmamba Added empty dependency for problem type SOLVER_RULE_UPDATE
warning  libmamba Problem type not implemented SOLVER_RULE_STRICT_REPO_PRIORITY
Could not solve for environment specs
The following packages are incompatible
├─ mpi4py is installable with the potential options
│  ├─ mpi4py [2.0.0|3.0.0|3.0.1|3.0.2|3.0.3] would require
│  │  └─ python [2.7* |>=2.7,<2.8.0a0 ], which can be installed;
│  ├─ mpi4py 2.0.0 would require
│  │  └─ python 3.4* , which can be installed;
│  ├─ mpi4py [2.0.0|3.0.0] would require
│  │  └─ python [3.5* |>=3.5,<3.6.0a0 ], which can be installed;
│  ├─ mpi4py [2.0.0|3.0.0] would require
│  │  └─ python 3.6* , which can be installed;
│  ├─ mpi4py 3.0.0 would require
│  │  └─ openmpi 3.0.* , which does not exist (perhaps a missing channel);
│  ├─ mpi4py [3.0.0|3.0.1|...|3.1.1] would require
│  │  └─ python >=3.6,<3.7.0a0 , which can be installed;
│  ├─ mpi4py [3.0.0|3.0.1|...|3.1.3] would require
│  │  └─ python >=3.7,<3.8.0a0 , which can be installed;
│  ├─ mpi4py [3.0.3|3.1.0|...|3.1.4] would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  ├─ mpi4py [3.0.3|3.1.0|...|3.1.4] would require
│  │  └─ python >=3.9,<3.10.0a0 , which can be installed;
│  ├─ mpi4py [3.1.1|3.1.2|3.1.3|3.1.4] would require
│  │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  ├─ mpi4py [3.1.3|3.1.4] would require
│  │  └─ mpich >=4.0.2,<5.0a0 , which can be installed;
│  ├─ mpi4py [3.1.3|3.1.4] would require
│  │  └─ openmpi >=4.1.4,<5.0a0 , which requires
│  │     └─ mpi 1.0 openmpi, which can be installed;
│  └─ mpi4py [3.0.3|3.1.4] conflicts with any installable versions previously reported;
└─ mpich 3* external_* is not installable because there are no viable options
   ├─ mpich [3.3.1|3.3.2|...|3.4.3] would require
   │  └─ mpi 1.0 mpich, which conflicts with any installable versions previously reported;
   └─ mpich 3.3.2 conflicts with any installable versions previously reported.

[dalcinl]

I'm puzzled. What's going on here? Is is somehow mpi4py-feedstock's fault?

[bernstei]

I don't know if I'd call it a "fault", but (assuming it's as simple as building a new wheel) it'd be nice to have conda's mpi4py work with the latest default python version.

[bernstei]

Going to python 3.10 allowed mpi4py to install in my conda env with the external mpich package.

Sadly, ldd still reported the centrally installed libmpi.so.12 for my mpi4py's MPI.cpython*.so, but then I got suspicious that maybe the HPC's environment modules that enable conda are breaking something with my PATH or LD_LIBRARY_PATH. I stopped using the HPC-provided conda environment module (module load cse/anaconda3), and instead just got conda set up by running the profile.d/conda.sh script manually. However, after installing mpi4py with mamba and the external mpich, it still didn't work. The problem is that, for some reason, the version of libmpi.so that is dynamically linked to the MPI.cpython....so is libmpi.so.12, and that's not the same version number as the HPC-provided version, so it refused to use the HPC one and just showed a failed dependency.

I then tried with openmpi, and it appears to work at last - at least ldd on the mpi4py so file reports the system (not conda) mpi library. I would have preferred to use mpt because it generally gets better performance on this machine, but it's definitely better than nothing. I'll report soon on whether it actually runs.

[bernstei]

Well, openmpi didn't work, probably because despite being installed it wasn't really supported, I suspect. I did get intelmpi, which is based on mpich and luckily uses the same libmpi.so.12 filename, to work, by uninstalling openmpi and mpi4py and going back to the external mpich version (+ mpi4py).

Is it expected that the number version of the libmpi.so dependency is hardwired into the MPI.cpython....so file that's installed as part of mpi4py? That strikes me as a very inconvenient restriction.

[leofang]

Sorry guys, was traveling and couldn't respond.

I'm puzzled. What's going on here? Is is somehow mpi4py-feedstock's fault?

Kinda. Point being, we somehow only built mpi4py + mpich 3 up to Python 3.10; when the PY311 migrator came, we already moved to mpich 4 and never looked back. It seems to me we need to expand our build matrix so as to serve, say, the most recent 2 MPI versions, but let's not discuss here. I'll open a new issue in mpi4py-feedstock shortly.

Is it expected that the number version of the libmpi.so dependency is hardwired into the MPI.cpython....so file that's installed as part of mpi4py? That strikes me as a very inconvenient restriction.

Yes, as Lisandro explained below, the version info is hardwired by the linker at the compile time. There's nothing we can do for such ABI issues, unless

1. the MPI-5 standard lands, or
2. we support a wide range of build matrix in conda-forge (this might be frowned upon).

In any case, the discussion for conda-forge packages should be moved to elsewhere.

[dalcinl]

@bernstei What happens if you just remove mpi4py and any MPI package from your environment, and then you just pip install mpi4py? I'm assuming your HPC system mpicc command can be found in your $PATH. If it fails, can you show us the output?

EDIT: Also what version of MPICH do you have in your HPC system?

[bernstei]

As for versions, I don't have a real MPICH. I have mpt 2.21 (by default - others are available), openmpi 3.x, and intelmpi (which I believe is based on mpich, but I don't know what version) 18.0.3.222.

[bernstei]

Aha - now it appears to work, at least at the level of ldd MPI.cpython*.so. The mpi4py built by pip looks for libmpi.so, and then it successfully finds any mpi I have loaded (via an env module that sets LD_LIBRARY_PATH, I guess), including mpt, intelmpi, and openmpi (because they all provide libmpi.so, in addition to libraries with specific version numbers). I guess it must have been some path mangled by the HPC-provided anaconda env module that broke the original pip install process.

So the answer to my question is to make sure that the PATH (or LD_LIBRARY_PATH) isn't broken, create a conda venv with no MPI installed, and then pip install mpi4py.

I really appreciate your help figuring this out. It's always a pain trying to work through any given idiosyncratic HPC configuration.

[bernstei]

I guess it would be nice if the conda-provided mpi4py also created a MPI.cpython*.so that had no particular libmpi.so library version requirement, but for me at least pip is a good solution.

[dalcinl]

I guess it would be nice if the conda-provided mpi4py also created a MPI.cpython*.so that had no particular libmpi.so library version requirement, but for me at least pip is a good solution.

IMHO, that's not a good idea. MPIs are not interchangeable. I'm working with other folks to develop an MPI-ABI for the upcoming MPI-5 standard. Without a common ABI, you cannot just switch MPI providers at runtime. Within a same family of MPI implementations (e.g. MPICH and its derivative Intel MPI), things are expected to work. The problem is that your Intel MPI is quite dated, and its library is most likely not compatible with the MPICH v4.x used to build mpi4py on conda-forge. In those nasty situations, pip install mpi4py comes to rescue at the price of waiting a few minutes for the package to build. But PLEASE MAKE SURE you run mpi4py with the same MPI implementation you used to build it, otherwise things will fail badly.

[bernstei]

OK, that's good to know. I'm happy to stick to the pip compiled one with a matching (real) mpicc, now that I figured out how to get conda's python to not use any of the central conda libraries.

[dalcinl]

Is it expected that the number version of the libmpi.so dependency is hardwired into the MPI.cpython....so file that's installed as part of mpi4py? That strikes me as a very inconvenient restriction.

It is, it is totally expected, as it is the SONAME MPICH uses for its libraries. It would be a very bad idea to change it.

There is trivial workaround, though. I cannot assert it would work under complicated scenarios involving multiple packages, but you can try at your own risk. The workaround is as follow (you can use a different folder than your $HOME, I just use it for convenience):

let LIBMPI_SO=/path/to/mpi_root/lib/libmpi.so
mkdir -p $HOME/lib
ln -s $LIBMPI_SO $HOME/lib/libmpi.so.12
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/lib

[bernstei]

The real issue is ABI compatibility, right? I guess I didn't expect that the number version would be a sufficiently reliable indicator of ABI compatibility to be useful.

Anyway, thanks for the suggestion - I'll try it next time I need to test a different MPI version.

[dalcinl]

It is not only an issue of ABI compatibility. ABI compatibility is a must, but also the SONAMEs have to much, otherwise the dynamic linker gets confused. For some reason, the person that setup your system decided to call (I mean, SONAME) the MPI library just libmpi.so? What's the output you get if you run readelf -d /path/to/your/mpi/lib/libmpi.so ?

[bernstei]

This is a standard HPE (formerly SGI, I guess, since I believe mpt was inherited from SGI as per https://downloads.linux.hpe.com/SDR/project/mpi/) installation in this case, so while it may or may not be a good idea, I don't think it's a random person making a personal decision.

$ readelf -d /p/app/hpe/mpt-2.21/lib/libmpi.so 

Dynamic section at offset 0x1d9eb8 contains 29 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libcpuset.so.1]
 0x000000000000000c (INIT)               0x6aff0
 0x000000000000000d (FINI)               0x1961a0
 0x0000000000000019 (INIT_ARRAY)         0x3d9b00
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x3d9b08
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x228
 0x0000000000000005 (STRTAB)             0x1b288
 0x0000000000000006 (SYMTAB)             0x65a0
 0x000000000000000a (STRSZ)              45124 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x3db000
 0x0000000000000002 (PLTRELSZ)           21144 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x2f8f8
 0x0000000000000007 (RELA)               0x27f90
 0x0000000000000008 (RELASZ)             31080 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x27e90
 0x000000006fffffff (VERNEEDNUM)         5
 0x000000006ffffff0 (VERSYM)             0x262cc
 0x000000006ffffff9 (RELACOUNT)          674
 0x0000000000000000 (NULL)               0x0

[dalcinl]

Oh, but then this is not even an MPICH derivative, IIRC. So no ABI compatibility to start with. Then there is no way you can use MPI conda-forge packages. pip install mpi4py is your only friend. BTW, did you run in any issues with that?

PS: I used to test in the MPI implementation you are using. But at some point it stopped being available for download. Therefore I stopped caring about it.

[bernstei]

You may be right about it not being MPICH-derived.

I'm current running happily with intel MPI, which I think is an MPICH derivative, using the pip install (rather than the conda mpich variant), once I stopped using the env module they provide and instead doing my own conda initialization. I did minimal tests with mpt and they seemed to work, but by then I already had intel-mpi based runs working fine, so I didn't pursue it further. And this machine is being decommissioned in a few months, so soon it'll become moot.