mpi4py: Run mpi4py tests with a relocated Open MPI installation

[dalcinl]

Addressing #12349 (comment).

[github-actions]

Hello! The Git Commit Checker CI bot found a few problems with this PR:

1ca576e: mpi4py: Run mpi4py tests with a relocated Open MPI...

• check_signed_off: does not contain a valid Signed-off-by line

1163918: mpi4py: Support for workflow_dispatch trigger

• check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

[dalcinl]

@wenduwan An obvious problem is that Spawn is still broken for other reasons. Even if you add the label to run the spawn tests, I believe these will fail before we reach the point where relocation is tested.

[wenduwan]

bot:aws:retest

[wenduwan]

We already have default value for inputs.repository so should we drop || 'mpi4py/mpi4py'?

[dalcinl]

That default applies to the workflow_dispatch trigger to run the workflow via the GitHub Web UI. For the case of the pull_request trigger, these input values are null and eventually evaluate to an empty string. Therefore, to the best of my knowledge, we need the || 'mpi4py/mpi4py'.

[rhc54]

@wenduwan An obvious problem is that Spawn is still broken for other reasons. Even if you add the label to run the spawn tests, I believe these will fail before we reach the point where relocation is tested.

I'm unsure what error you are referring to here. I tested singleton spawn, and that worked fine. I then ran your loop spawn program for 500 iterations, testing it with 1-3 processes and spawning 1-3 processes - and that worked fine. I did have to allow oversubscription when combining 3 procs with spawning 3 procs, or else it would run out of slots after 3 iterations, so it looks like there might be a race condition on recovering resources. Still, with that caveat, spawn is working.

Is there something else that is wrong?

[dalcinl]

I tested singleton spawn,

Indeed, the spawn tests run in isolation of other tests seem to be working fine.
Did you finally merge these changes related to not using accept/connect and move to something more robust?

Is there something else that is wrong?

Yes, look at the failing mpi4py/run_spawn/mpi4py-tests results [link]. After running all the spawn tests, looks like ompi's internal state ends somehow broken, and attempting to create an intercommunicator fails with MPI_ERR_INTERN.

[dalcinl]

@rhc54 The following command line is the minimal set of tests to trigger the issue:

env MPI4PY_TEST_SPAWN=1 mpiexec -n 2 python test/main.py -v test_comm test_spawn test_ulfm

I'm seeing the following lines in the output:

[kw61149:364081] [[389,1],1] selected pml ob1, but peer [[389,1],0] on kw61149 selected pml ��
[kw61149:364081] OPAL ERROR: Unreachable in file ../../../ompi-main/ompi/communicator/comm.c at line 2372
[kw61149:364081] 0: Error in ompi_comm_get_rprocs
setUpClass (test_ulfm.TestULFMInter) ... ERROR

[rhc54]

Did you finally merge these changes related to not using accept/connect and move to something more robust?

Afraid not - been off doing other things, but am returning to these issues now. No timeline for completion.

[kw61149:364081] [[389,1],1] selected pml ob1, but peer [[389,1],0] on kw61149 selected pml ��

Odd - I'm pretty sure we fixed that one. It took me some time and help from a user that could reproduce it to track it down, and the fix went into PMIx master two weeks ago.

Checking the state of OMPI's PMIx submodule, it is (a) out-of-date by more than a month, and therefore missing that fix, and (b) in some odd detached state. It's almost like someone cherry-picked some change into it rather than updating the submodule pointer?

Anyway, the problem is indeed fixed - just not over there for some reason.

[janjust]

I'll push a submodule update to this PR shortly

[wenduwan]

@janjust do you have bandwidth to update this PR? I can do it otherwise.

[wenduwan]

Turns out I don't have permission to push to the PR. Opened #12532 and will rebase this one.

[wenduwan]

Howard leapfrogged me with #12565 will likely merge that instead.