Oversubscribe

[Chrismarsh]

When launching a process via MPI.COMM_SELF.Spawn how can one oversubcribe the system?

[dalcinl]

I'm afraid I do not understand your question. mpi4py does not deal with oversubscription in any way. It is the backend MPI implementation that handles the details, and whether oversubscription is allowed or not. From mpi4py's side, you just ask to Spawn how many processes you want/need (maxprocs argument of Spawn).

[Chrismarsh]

My understanding is that calling MPI.COMM_SELF.Spawn('/usr/bin/some_process', maxprocs=4) from within a python script is equivalent to mpirun -np 4 /usr/bin/some_process.

I would like the equivalent of mpirun -np 4 --oversubscribe /usr/bin/some_process when called from MPI.COMM_SELF.Spawn. e.g., something like

MPI.COMM_SELF.Spawn('/usr/bin/some_process', maxprocs=4, oversubscribe=true).

Reposed, if I have code like:

$ python my-code.py

where my-code.py spawns MPI processes via MPI.COMM_SELF.Spawn, then I can oversubscribe it on OpenMPI via

$ export OMPI_MCA_rmaps_base_oversubscribe=true; python my-code.py

and I would like to do this from within the python code in a more generic fashion.

Does that clarify sufficiently?

[dalcinl]

Yes, now everything is clear. Unfortunately, what you ask is not under mpi4py's control. You'll have to ask Open MPI folks (or dive in docs/source code) to see if they support passing Spawn() a hint via Info object to enable oversubscription. Alternatively, you could just enable oversubscription permanently in your system with by adding the line rmaps_base_oversubscribe=true to ~/.openmpi/mca-params.conf, like I do here. Note that for Open MPI v5.0.x thing have changed slightly, but the config from the previous link should work for both v4.1.x and v5.0.x.

[Chrismarsh]

I really only have experience with OpenMPI. From your answer, I take it that different MPI's mpirun/mpiexec take different arguments for if/how they can oversubscribe?

Is it possible to pass envvars through to Spawn? Alternatively, is the env of Spawn inherited from the calling python such that os.setenv(...) would be visible to the MPI ranks?

[dalcinl]

different MPI's mpirun/mpiexec take different arguments for if/how they can oversubscribe?

Yes. MPICH and its derivatives (like Intel MPI) usually allow oversubscription, but performance will suck unless you use environment variables to tweak the MPI runtime to not use aggressive polling on wait/progress operations.

Is it possible to pass envvars through to Spawn?

Yes, it may be supported via the Info hints. However, that would set the environment for the new MPI processes, and not necessarily affect oversubscription.

Alternatively, is the env of Spawn inherited from the calling python such that os.setenv(...) would be visible to the MPI ranks?

Usually yes. Again, all that is under the control of the backend MPI implementation. Even if you os.setenv right before Spawn, that may not be enough to affect oversubscription behavior, Open MPI may read the OMPI_MCA_rmaps_base_oversubscribe=true setting at MPI initialization time, and not at Spawn time.

[Chrismarsh]

Gotcha, that's very clear. I will just advise my users to set the envar directly or to set the config file as you suggest. Thank you.