Using mpi4py.futures within Jupyter notebooks

[YarShev]

Hi there,

I would really like to know how to fix or work around the problem I encountered when running a simple code snipper from the docs in a jupyter notebook.

from mpi4py import MPI
from mpi4py.futures import MPICommExecutor

with MPICommExecutor(MPI.COMM_WORLD, root=0) as executor:
    if executor is not None:
       future = executor.submit(abs, -42)
       assert future.result() == 42
       answer = set(executor.map(abs, [-42, 42]))
       assert answer == {42}

When I run this code with

MPI4PY_FUTURES_MAX_WORKERS=8 mpiexec -n 1 jupyter notebook

in a jupyter notebook, I get the error The kernel appears to have died. It will restart automatically.. The problem happens on Linux, but on Windows it works fine.

Thanks in advance.

[dalcinl]

I see you are setting MPI4PY_FUTURES_MAX_WORKERS=8. But then executing jupyter with just one MPI process. Therefore, you should not use MPICommExecutor, you should use instead MPIPoolExecutor to get mpi4py.futures spawn the 8 worker processes you want.

Using mpi4py under Jupyter is a bit tricky, mostly because MPI implementations sometimes do not support spawning new processes (the 8 workers you want) if the parent process (the kernel) was not executed via mpiexec. Note that you are using mpiexec indeed, but you are using it to start Jupyter, and that does not mean Jupyter will start the kernel process with mpiexec. Things may not work.

Alternatively, you may setup Jupyter to start a cluster using the MPI engine. This cluster needs just one process (the parent), then mpi4py.futures will take care of spawning the workers via standard MPI mechanisms. Check Jupyter docs about all that stuff. Unfortunately, I'm not a savvy user of parallel IPython/Jupyter, sorry I cannot provide more useful help.

[YarShev]

Oh, now I see the difference between MPICommExecutor and MPIPoolExecutor. The former just splits the set of MPI processes within a MPI communicator in one master process and many workers processes. However, I am still experiencing the same behavior even with MPIPoolExecutor on both Linux (it doesn't work) and Windows (it works). I tried this script.

from mpi4py import MPI
from mpi4py.futures import MPIPoolExecutor

if __name__ == "__main__":
    with MPIPoolExecutor() as executor:
        if executor is not None:
           future = executor.submit(abs, -42)
           assert future.result() == 42
           answer = set(executor.map(abs, [-42, 42]))
           assert answer == {42}

Why doesn't the use of mpiexec mean Jupyter will start the kernel process with mpiexec?

Regarding the set up of jupyter to start a cluster using the MPI engine, I will try to find the docs and go through it.

Meanwhile, could you please tag some more people who may be aware of this problem and of its resolving?

[dalcinl]

Why doesn't the use of mpiexec mean Jupyter will start the kernel process with mpiexec?

If I understand things correctly, the kernel runs in a separate process created at some point, and I don't think that process is under mpiexec control.

Meanwhile, could you please tag some more people who may be aware of this problem and of its resolving?

I don't really know. Maybe @leofang can think of someone?

PS: What MPI implementation are you using on Linux and Windows?

[leofang]

Regarding the set up of jupyter to start a cluster using the MPI engine, I will try to find the docs and go through it.

IIUC you can refer to these resources (disclaimer: I am on my cell and I didn't try)

• https://ipyparallel.readthedocs.io/en/latest/reference/mpi.html
• https://charlesreid1.com/wiki/Jupyter/MPI#Start_MPI_Cluster

I think the key is to ensure whenever Jupyter launches a kernel, it's launched under mpiexec. Then the kernel can spawn new processes as long as your MPI backend supports it.

Meanwhile, could you please tag some more people who may be aware of this problem and of its resolving?

I don't really know. Maybe @leofang can think of someone?

IIRC @minrk is the expert? 🙂

[YarShev]

Why doesn't the use of mpiexec mean Jupyter will start the kernel process with mpiexec?

If I understand things correctly, the kernel runs in a separate process created at some point, and I don't think that process is under mpiexec control.

That might be the case because I see three new processes (mpiexec, hydra_pmi_proxy, jupyter-noteboo) in top on Linux and two new processes (mpiexec.exe, jupyter.exe, but there is no hydra_pmi_proxy.exe) in tasklist on Windows when I run.

mpiexec -n 1 jupyter notebook

PS: What MPI implementation are you using on Linux and Windows?

Windows

mpi                       1.0                       msmpi    conda-forge
mpi4py                    3.1.4            py38h73c715a_0    conda-forge
msmpi                     10.1.1               h3502643_7    conda-forge

Linux

mpi                       1.0                       mpich    conda-forge
mpi4py                    3.1.4            py38h97ac3a3_0    conda-forge
mpich                     4.0.3              h846660c_100    conda-forge

[YarShev]

Regarding the set up of jupyter to start a cluster using the MPI engine, I will try to find the docs and go through it.

IIUC you can refer to these resources (disclaimer: I am on my cell and I didn't try)

• https://ipyparallel.readthedocs.io/en/latest/reference/mpi.html
• https://charlesreid1.com/wiki/Jupyter/MPI#Start_MPI_Cluster

I think the key is to ensure whenever Jupyter launches a kernel, it's launched under mpiexec. Then the kernel can spawn new processes as long as your MPI backend supports it.

I will go through these resources. They probably explain how to configure jupyter in order for it to run a kernel under mpiexec.

IIRC @minrk is the expert? 🙂

I would be really appreciate any help 🙂.

[dalcinl]

Oh, I see... you are using MPICH. That explains why your kernel process dies. As the kernel process is not executed with mpiexec, the MPI spawn call to create worker process when using MPIPoolExecutor fails. That's a know issue with the MPICH version you are using. Fortunately, next MPICH release 4.1 should work just fine.

@hzhou This is precisely the use case I've been so insistent in the past about getting singleton init + spawn working seamlessly. Thanks again for your prompt action on fixing this long standing issue.

[YarShev]

@dalcinl, do you mean that starting MPICH release 4.1 it should be able to run mpi4py in a jupyter notebook with the command I posted and without any additional setup?

[dalcinl]

Yes, exactly. Actually, you will not even need to execute jupyter with mpiexec. Things should just work out of the box.

[YarShev]

That would be great! Look forward to trying the release 4.1.

[minrk]

I think the key is to ensure whenever Jupyter launches a kernel, it's launched under mpiexec

To start a kernel process with mpiexec, you'll want to modify kernel.json and add mpiexec to the command launched (argv). You can locate the kernelspec directory with jupyter kernelspec list, but it will generally be at $PREFIX/share/jupyter/kernels/python3/kernel.json. You can change the command to be a more customizable launch script, rather than launching ipykernel directly, if you want to be able to e.g. try to detect whether mpiexec should be invoked or not.

You may also want to copy the python3 kernel to a new directory, e.g. mpi (this will create an 'mpi' kernel in your kernel list), if you want to be able to pick mpi or not on a per-notebook basis.

[dalcinl]

After upgrading to Py 3.10 and latest mpi4py 3.1.4, things get even worse, the recv_mprove hack no longer works. Long story short, IMHO, Intel MPI is broken. As an occasional user with no access to Intel MP source code, I'm not in a position to investigate this further.

[YarShev]

Thank you very much for the investigation! This is very helpful for me to move forward. Could you please elaborate the issue with rc.recv_mprobe or point me out at some reference discussion? Say If I want to use MPI probing in terms of performance, why I can't use it with mpi4py (I have to disable it to make things work)?

[YarShev]

Btw, I see only comm.Probe method in the mpi4py documentation - https://mpi4py.readthedocs.io/en/stable/reference/mpi4py.MPI.Comm.html?highlight=probe#mpi4py.MPI.Comm.Probe. Why there is no like comm.MProbe as in C?

[dalcinl]

Could you please elaborate the issue with rc.recv_mprobe or point me out at some reference discussion?

Intel had a webpage, but looks like it is gone by now. That page simply mentioned the workaround I explained above, but the Intel folks never really addressed the root of the problem. Many years after, I'm are still there advising people about it, it is really frustrating.

why I can't use it with mpi4py (I have to disable it to make things work)?

It is not about performance. The actual problem is that some codes eventually deadlock.

Why there is no like comm.MProbe as in C?

Simply because there is no 'MProbe' in C, but Mprobe instead. Just look in the webpage you linked for the entry right above Probe.

PS: I converted your original issue as a conversation to mark this answer as accepted. Your original question was about MPICH. Now we are digressing to address Intel MPI. We should stop adding to this thread to avoid confusion for future readers. Feel free to open a new issue, or let's continue via email in mpi4py@googlegroups.com

[YarShev]

Yes, I think we duscussed enough here. Thank you very much.