Process pinning with MPIPoolExecutor

[FabioLuporini]

Hello!

Question: what's the recommended way of pinning the worker processes when using an MPIPoolExecutor?

Context. My user runs a Python application that calls into a function provided by my library.

<some code>

output = blocking_library_call()

<some other code>

The user has zero vision whatsoever of MPI. He/she will run the app as python code.py, that is, without using mpirun. This app only runs on the local node.

Underneath, blocking_library_call spawns an MPIPoolExecutor (which persists across multiple calls potentially, but that's a different story) which executes one or more tasks. In fact, the MPI workers will collectively execute a data-parallel application (with actual sends and receives buried inside). In practice I will have as many MPI "workers" as NUMA domains available on the underlying architecture, and I want to pin them such that there's one MPI worker per NUMA region. This is trivial to do if I ran the code as mpirun -n 4 <mpi-distro-specific-binding-options> python code.py, but that's not my case.

I tried setting an env var, e.g. OMPI_MCA_hwloc_base_binding_policy=numa python code.py (this is with OpenMPI) but the process crashes immediately with the following error:

--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  Setting processor affinity failed failed
  --> Returned value Error (-1) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[a75b40a357db:04485] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

And lastly, based on what I read around including the documentation, MPI_Info doesn't appear to be the right mechanism to achieve this, though I may be wrong.

So basically I don't know how to do this or if this is a current limitation of mpi4py (in which case I might be willing to contribute depending on the amount of effort required) or something else.

Sort-of related question: the process spawning the MPI pool at some point performs results = executor.map(...); I then block immediately by waiting for the results to be available. During this wait, does the process busy wait (thus consuming a physical core) or does it go sleep? I guess the latter, but just to be sure

Thanks a lot for reaching the bottom of this issue. Really appreciate your invaluable work for entire open-source / computational science community (I've been using mpi4py for > 10 years)

[dalcinl]

Question: what's the recommended way of pinning the worker processes when using an MPIPoolExecutor?

You should look at the documentation of the backend MPI library about whether they support passing pinning hints to spawned processes, for example, maybe via the MPI_Info argument. I think the info argument would be an ideal mechanism to specify these options. But it is up to implementations to support such mechanism. If you use Open MPI, you should ask Open MPI folks.

Are you aware about using mpiexec ... python -m mpi4py.futures ... to launch master and workers in advance, and let mpi4py transparently split WORLD in one master and many workers, without any spawning involved? It is admittedly not ideal, but you can control pinning the usual way.

The user has zero vision whatsoever of MPI.

Isn't just great to hide so much mess and nonsense from users? 🤣

So basically I don't know how to do this or if this is a current limitation of mpi4py

Why would you even consider there is a limitation is in mpi4py? How would you achieve pinning in a pure-C application using MPI spawning?

I tried setting an env var, e.g. OMPI_MCA_hwloc_base_binding_policy=numa python code.py (this is with OpenMPI) but the process crashes immediately with the following error.

Ask Open MPI folks, please. I've spent hours and hours over the last couple years setting up automated testing to report regressions to them, and despite some improvements, the whole spawn thing is still unreliable, currently broken in the v5.0.x branch. I have 99.99% certainty that any issue you may experience is not mpi4py's fault, and unless proven incorrect, there's nothing I can do.

Sort-of related question: the process spawning the MPI pool at some point performs results = executor.map(...); I then block immediately by waiting for the results to be available. During this wait, does the process busy wait (thus consuming a physical core) or does it go sleep? I guess the latter, but just to be sure

If I had coded things the obvious MPI way, and given the default way MPI implementation works, the waiting would have been a 100% CPU busy wait. MPI implementations do not care about your CPU burning, they care about beating each other a latency contests. However, your CPU burning will not be what happens by the default when using mpi4py.futures. I you let me get technical, what mpi4py does is a recurrent MPI polling with exponential backoff (details in the source code).

[FabioLuporini]

Isn't just great to hide so much mess and nonsense from users? 🤣

I know right! 😂 😂

Why would you even consider there is a limitation is in mpi4py?

I don't know honestly I just admit my ignorance 😬 first time I try to use the dynamic spawning mechanism and the boundary between the underlying mpi and mpi4py is still a bit blurred :)

I'll get in touch with the OpenMPI team to see what they have to say, if you don't mind leaving this discussion open for the time being I'll be back with explanation for future readers

Thanks for the great reply!

[dalcinl]

Note that I migrated you original issue to a discussion in the Q&A section. It will be kept open for ever.

[dalcinl]

@mrogowski Did we ever figure out a way to do pinning of spawned processes when you were benchmarking for our mpi4py.futures paper?

[mrogowski]

No. I believe we used MPICH and -bind-to hwthread -map-by core when running in the "legacy" mode (i.e., mpiexec ... python -m mpi4py.futures ... launching all the processes in advance) and no binding when dynamically spawning processes. The justification was partially to make for a fair comparison with concurrent.futures.

[FabioLuporini]

I found what I needed, tested it, and I think it works.

So according to the OpenMPI documentation I can pass bind_to and map_by to MPI_Info:

https://www.open-mpi.org/doc/v4.1/man3/MPI_Comm_spawn.3.php

so all I had to do was:

+        mpi_info = MPI.Info.Create()
+        mpi_info['bind_to'] = 'numa'
+        mpi_info['map_by'] = 'numa'
+
+        executor = MPIPoolExecutor(max_workers=nranks, mpi_info=mpi_info)

and it seems like I'm getting what I needed

Thanks!

[dalcinl]

Remember to info.Free() after creating the executor to not leak the MPI-level MPI_Info handle. Otherwise, try passing mpi_info as a plain Python dict object, IIRC the thing will be converted to Info under the hood (and properly disposed when no longer needed).

[FabioLuporini]

also, about your earlier comment:

If I had coded things the obvious MPI way, and given the default way MPI implementation works, the waiting would have been a 100% CPU busy wait. MPI implementations do not care about your CPU burning, they care about beating each other a latency contests. However, your CPU burning will not be what happens by the default when using mpi4py.futures. I you let me get technical, what mpi4py does is a recurrent MPI polling with exponential backoff (details in the source code)

I think I kinda disagree here with what you might have wanted to do. I think this:

what mpi4py does is a recurrent MPI polling with exponential backoff (details in the source code)

is actually quite sensical, at least for the use I'm making of this mechanism, which is admittedly oriented towards data-parallelism rather than task-parallelism. Because in my case, which yeah might slightly bend the original intent of the MPIPoolExecutor, I want all of the physical cores at my disposal for a signficant amount of time to execute a data parallel numerical simulation. So I'm happy the master process doesn't get too much in the way.

Perhaps this could be controllable with an option to specify the polling time (defaulting to what you have in place if left unspecified, and behaving like a busy wait if set to 0)

Anyway, I'm guilty of diverging here :)

Again, thanks a lot for the prompt answer. Really appreciated it 👍

[dalcinl]

Perhaps this could be controllable with an option to specify the polling time (defaulting to what you have in place if left unspecified, and behaving like a busy wait if set to 0)

As I said before, the details are all in the source code.
Indeed, you can set the env var MPI4PY_FUTURES_BACKOFF=0.001 to get a 1ms delay (BTW, that's the default value), or set it to just 0 for busy waiting. You could also pass backoff=0.001 as a kwarg when creating the executor.
And the next step is you complaining about this not being documented... Please don't go there 😄.
This is a tweak for professionals that I'm not fully committed to keep in that exact way forever, therefore I prefer to not advertise it too much and have the freedom to do whatever I see fit in the future. Anyway, I hope you will trust me by now I'll not do nonsensical things and I'll keep everyone's back covered.