"Failed to modify UD QP to INIT on mlx5_0: Operation not permitted" error with OpenMPI after updating to OFED 23.10

[boegel]

After updating to OFED 23.10, we had several reports of people running this cryptic error:

Failed to modify UD QP to INIT on mlx5_0: Operation not permitted

The problem does not occur consistently (@hajgato has more details here).

openucx/ucx#9468 suggests that updating to a more recent OFED may help, but it doesn't.

[boegel]

We've pinned this down to the libfabric dependency we include with OpenMPI: when libfabric is removed as a dependency, the problem no longer occurs.

This requires rebuilding OpenMPI, so that's painful on a production system.

As a workaround, you can instruct OpenMPI to not use libfabric by passing the following options to mpirun:

mpirun -mca pml ucx -mca btl '^uct,ofi' -mca mtl '^ofi'

Or equivalently, you can set the following environment variables:

export OMPI_MCA_btl='^uct,ofi'
export OMPI_MCA_pml='ucx'
export OMPI_MCA_mtl='^ofi'

[boegel]

SURF (@casparvl) also saw a very similar issue, they worked around it by setting $FI_PROVIDER to verbs...

The error there was Invalid argument though, more similar to what was reported in openucx/ucx#9468

[branfosj]

I've not seen this, but we added the following in December 2021 because of another issue we'd seen

            # avoid libfabric warning "unknown link width 0x10"
            # see https://github.com/ComputeCanada/software-stack-config/pull/19
            setenv OMPI_MCA_mtl "^ofi"
            setenv OMPI_MCA_btl "^openib,ofi"

[casparvl]

Yep, we get these:

avoid libfabric warning "unknown link width 0x10"

As well regularly. We also solved it by setting those envrionment variables (I believe, would need to check what we set exactly)

[casparvl]

More detail on what I hit here btw