Conda dependencies broken by OpenMPI 4.1.3

[jhrmnn]

OpenMPI 4.1.3 was published on conda-forge yesterday, and it now has also the "external" variant which is supposed to be used with an external MPI library (conda-forge/openmpi-feedstock#95). This variant has a lower priority, so the loaded variants should get installed by default, but for some reason conda install 'dftbplus=*=*openmpi*' tries to pull the external variant as a dependency. The same happens to Libmbd. conda install openmpi works as expected. Not sure if it's our or their problem.

[awvwgk]

This might have been just the CDN delay on the anaconda cloud. Could you try whether this is still an issue today?

[awvwgk]

Can you try solving the environment with mamba instead (the conda solver tends to get stuck from time to time)?

❯ mamba create -n dftbplus2 'dftbplus=*=*openmpi*'

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (0.22.1) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████


Looking for: ['dftbplus=[build=*openmpi*]']

conda-forge/noarch                                 @ 382.5kB/s 20.8s
conda-forge/linux-64                               @ 396.6kB/s 57.0s
Transaction

  Prefix: /home/awvwgk/software/opt/conda/envs/dftbplus2

  Updating specs:

   - dftbplus=*[build=*openmpi*]


  Package                  Version  Build                   Channel                    Size
─────────────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex              0.1  conda_forge             conda-forge/linux-64     Cached
  + _openmp_mutex              4.5  1_gnu                   conda-forge/linux-64     Cached
  + chimes-calculator        1.0.0  h0c14de7_1              conda-forge/linux-64     Cached
  + dftbplus                  21.2  mpi_openmpi_haf9840c_2  conda-forge/linux-64       18MB
  + dftd4                    3.3.0  hf06ca72_3              conda-forge/linux-64     Cached
  + elpa               2021.11.001  mpi_openmpi_haf9840c_3  conda-forge/linux-64        6MB
  + elsi                     2.8.2  mpi_openmpi_hbbb7b4b_2  conda-forge/linux-64      952kB
  + fftw                    3.3.10  nompi_h77c792f_102      conda-forge/linux-64     Cached
  + gawk                     5.1.0  h7f98852_0              conda-forge/linux-64     Cached
  + gsl                        2.7  he838d99_0              conda-forge/linux-64     Cached
  + libblas                  3.9.0  14_linux64_openblas     conda-forge/linux-64     Cached
  + libcblas                 3.9.0  14_linux64_openblas     conda-forge/linux-64     Cached
  + libgcc-ng               11.2.0  h1d223b6_15             conda-forge/linux-64     Cached
  + libgfortran-ng          11.2.0  h69a702a_15             conda-forge/linux-64     Cached
  + libgfortran5            11.2.0  h5c6108e_15             conda-forge/linux-64     Cached
  + libgomp                 11.2.0  h1d223b6_15             conda-forge/linux-64     Cached
  + liblapack                3.9.0  14_linux64_openblas     conda-forge/linux-64     Cached
  + libmbd                  0.12.5  mpi_openmpi_h4fb355c_0  conda-forge/linux-64      854kB
  + libnegf                  1.0.3  mpi_openmpi_hbbb7b4b_2  conda-forge/linux-64      797kB
  + libnsl                   2.0.0  h7f98852_0              conda-forge/linux-64     Cached
  + libomm                   1.0.0  mpi_openmpi_hbbb7b4b_2  conda-forge/linux-64      164kB
  + libopenblas             0.3.20  pthreads_h78a6416_0     conda-forge/linux-64     Cached
  + libstdcxx-ng            11.2.0  he4da1e4_15             conda-forge/linux-64     Cached
  + libzlib                 1.2.11  h166bdaf_1014           conda-forge/linux-64     Cached
  + matrixswitch             1.0.0  mpi_openmpi_hbbb7b4b_2  conda-forge/linux-64      321kB
  + mctc-lib                 0.2.5  hc3ea6d6_0              conda-forge/linux-64      495kB
  + mpi                        1.0  openmpi                 conda-forge/linux-64        4kB
  + mpifx                      1.2  mpi_openmpi_hcbd752a_0  conda-forge/linux-64      983kB
  + ntpoly                   2.7.1  mpi_openmpi_h113680b_1  conda-forge/linux-64      748kB
  + openmpi                  4.1.2  hbfc84c5_0              conda-forge/linux-64        5MB
  + plumed                   2.7.3  mpi_openmpi_heb59b2d_0  conda-forge/linux-64        4MB
  + scalapack                2.2.0  h67de57e_1              conda-forge/linux-64        2MB
  + scalapackfx              1.0.2  mpi_openmpi_h7636f6f_1  conda-forge/linux-64      401kB
  + simple-dftd3             0.4.2  hb01ceae_1              conda-forge/linux-64     Cached
  + tblite                   0.1.0  hf06ca72_2              conda-forge/linux-64     Cached
  + toml-f                   0.2.3  hc3ea6d6_0              conda-forge/linux-64      664kB
  + xdrfile                  1.1.4  h7f98852_1              conda-forge/linux-64     Cached
  + zlib                    1.2.11  h166bdaf_1014           conda-forge/linux-64     Cached

  Summary:

  Install: 38 packages

  Total download: 40MB

[jhrmnn]

Hm. I can reproduce that Mamba wants to install OpenMPI 4.1.2. So that's also unexpected, because there is a proper newer version 4.1.3 in three build versions, 0 (old), 1 (current, external), and 101 (current, loaded). So 101 should be installed. Conda wants to install 1. Mamba doesn't want to install any of them.

[awvwgk]

This combination cannot be installed, forcing mamba to pick the latest openmpi which is not external and the latest DFTB+ leads to a solver error

Looking for: ['dftbplus=21[build=*openmpi*]', 'openmpi==4.1.3[build=h*]']

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache
Encountered problems while solving:
  - package dftbplus-21.1-mpi_openmpi_hbbb7b4b_0 requires libgfortran5 >=9.3.0, but none of the providers can be installed

While the conda solver opts for installing the external openmpi package to circumvent this, mamba downgrades openmpi, which seems like the better choice IMO.

Means we probably have to do some rebuilding of our packages (again).

[jhrmnn]

Ah, good find. This is openmpi's problem and it's been already reported. conda-forge/openmpi-feedstock#94

[jhrmnn]

The upstream problem has been fixed and Conda now pulls the correct 4.1.3 build.