NA OFI: inadvertent activation of psm3 instead of psm2

[carns]

Describe the bug

After upgrading libfabric (via external repo at https://github.com/mochi-hpc/mochi-spack-packages/tree/main/packages/libfabric) from 1.11.1 to 1.13.0, psm2 performance on the LCRC Bebop system dropped by an order of magnitude. Performance with 1.13.0 returns to normal if libfabric is compiled with an explicit --disable-psm3 argument. The libfabric package linked above now has an explicit, hardcoded --disable-psm3 configure argument for this reason. At the moment we are not running mochi code on any native psm3 environments.

To Reproduce

Reproducing with the margo-p2p-bw and margo-p2p-latency tests in https://github.com/mochi-hpc-experiments/mochi-tests/blob/main/perf-regression/bebop/margo-regression.sbatch .

The package list looks like this:

argobots@1.1
cereal@1.3.0
intel-mpi@2019.8
json-c@0.15
libfabric@1.13.0
mercury@2.0.1
mochi-abt-io@0.5.1
mochi-bake@0.6.3
mochi-bedrock@0.3
mochi-margo@0.9.4
mochi-ssg@0.4.5
mochi-thallium@0.9.1
ncurses@5.9
nlohmann-json@3.9.1
numactl@2.0.11
opa-psm2@11.2.185
openssl@1.0.2k
pmdk@1.9
spdlog@1.8.5
tclap@1.2.2
util-linux-uuid@2.36.2

Expected behavior

I didn't expect the psm3 provider being enabled to impact psm2 performance. Are we activating the former by accident, and if so is there a way to prevent it from happening?

[carns]

@marcvef heads up