Use Mamba to install CI build dependencies

[benjaminhwilliams]

Use Mamba in place of Conda, in the hope that it will allow us to set up the xfel_regression test environment without errors. At the moment, that stage of the Python 3.9 bootstrap CI is always failing.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.34%. Comparing base (d2f8e5f) to head (2959053).
Report is 30 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2622      +/-   ##
==========================================
- Coverage   78.40%   78.34%   -0.06%     
==========================================
  Files         610      611       +1     
  Lines       75322    75438     +116     
  Branches    10745    10772      +27     
==========================================
+ Hits        59057    59104      +47     
- Misses      14092    14150      +58     
- Partials     2173     2184      +11     

[benjaminhwilliams]

@phyy-nx, this was an attempt to get around the CI issue with running the xfel_regression tests in the Python 3.9 bootstrap build. Originally, when using Conda to install the dependencies, we got this error:

PackagesNotFoundError: The following packages are not available from current channels:

  - hdf5[version='<1.13.*']
  - gemmi=0.6.3.*

Now, using Mamba instead, we get a different error:

Looking for: ['bash', 'distro', 'git-lfs', 'mpi4py', 'openmpi', 'pandas']

Pinned packages:
  - python 3.9.*

Could not solve for environment specs
The following packages are incompatible
├─ mpich is installable and it requires
│  └─ mpi 1.0 mpich, which can be installed;
└─ openmpi is not installable because it requires
   └─ mpi 1.0 openmpi, which conflicts with any installable versions previously reported.

I can reproduce this more simply on Ubuntu 22.04 and RHEL8 when running

$ python bootstrap.py update base --python 3.9
$ mamba install -dp ./conda_base openmpi

So it's a problem with installing Open MPI, as required by xfel_regression, specifically on a Python 3.9 build. Python 3.10 and 3.11 are both OK and Python 3.12 isn't yet supported for DIALS installs.

@phyy-nx, is Open MPI actually used in the xfel_regression tests?

[Baharis]

How does it relate to the fact that the gemmi version is now unpinned and required to be >0.6.5, see #2618 ?

[ndevenish]

It merely removes it from the complaint:

PackagesNotFoundError: The following packages are not available from current channels:

  - hdf5[version='<1.13.*']

Gemmi being mentioned is a red herring, it’s a side-effect of the conda resolver giving unsatisfactory results (admittedly conda is much better now, than the hundreds of pages it used to dump out).

[phyy-nx]

Hi, mpi is used in lots of xfel_regression tests.

dials/.azure-pipelines/unix-build.yml

Line 114 in d706264

mamba install -y bash distro git-lfs mpi4py openmpi pandas

Could openmpi be dropped from this line? mpi4py will bring in either openmpi or mpich, depending on which satisfies the dependency chain, and either are fine for the xfel_regression tests.

That would also sync up with how we do regular cctbx.xfel builds, where neither openmpi or mpich is specified:
https://github.com/cctbx/cctbx_project/blob/master/xfel/conda_envs/psana_environment.yml#L44

[benjaminhwilliams]

Could openmpi be dropped from this line? mpi4py will bring in either openmpi or mpich, depending on which satisfies the dependency chain, and either are fine for the xfel_regression tests.

Sorry, I missed this when you first posted it. Let's give it a go.