MPIPoolExecutor hanging in threading

[mloubout]

This is a follow-up/continuation of the work discussed in #476

ISSUE

We (I mainly) are running into an issue on some system that the executor hangs at either result fetching

File "/home/mloubout/.local/lib/python3.11/site-packages/mpi4py/futures/pool.py", line 234, in result_iterator
    yield futures.pop().result()

or at pool shutdown around here

Reproducer

The script hanging is:

from mpi4py.futures import MPIPoolExecutor
import threading
import time

def task(x):
    print(f"Processing {x}")
    time.sleep(1)  # Simulate a task taking some time
    return x * x

def main():
    with MPIPoolExecutor(max_workers=4) as executor:
        results = executor.map(task, range(10))
        print(list(results))

if __name__ == "__main__":
    main()

This is where it gets a bit tricky as it doesn't seem to always be the case. On my ubuntu desktop:

• it hangs with python 3.11 and 3.12 and mpi4py 3.1.5 and 3.1.6
• it runs fine within a docker image with python 3.10

The question

Are you aware of some interaction between mpi4py futures and threading that might lead to some deadlock?

Thanks again for all the support and discussions

[dalcinl]

@mloubout The issue may not be threading, but the backend MPI implementation. You did not mention what backend MPI implementation and version are you using. Is it Open MPI? Have you tried with a different MPI implementation?

Does it fail if you run your code above with mpiexec -n 5 python -m mpi4py.futures script.py ?

Would you mind doing a quick experiment?
Look at the following two lines in the mpi4py.futures sources:
https://github.com/mpi4py/mpi4py/blob/maint/src/mpi4py/futures/_lib.py#L578
https://github.com/mpi4py/mpi4py/blob/maint/src/mpi4py/futures/_lib.py#L658
Could you replace comm.Disconnect() by comm.Free() and tell us how it goes?

[mloubout]

u did not mention what backend MPI implementation and version are you using. Is it Open MPI?

I have tried two: Openmpi and intelMPI and they both hang.

Does it fail if you run your code above

The python3 -m mpi4py.futures script.py runs fine, it seems to hang only without the -m. The "little" issue is that what we are developing needs to be called from another program (julia) that doesn't have the capability to call python that way 😢

Could you replace comm.Disconnect() by comm.Free() and tell us how it goes?

Yes. It doesn't change the hang with openMPI or intel MPI

For reference, this is the error trace when I exit it (ctrl+c)

^C^CTraceback (most recent call last):
  File "/home/mloubout/DevitoCodes/devitopro/mfes/decoupler.py", line 13, in main
    print(list(results))
          ^^^^^^^^^^^^^
  File "/home/mloubout/.local/lib/python3.11/site-packages/mpi4py/futures/pool.py", line 234, in result_iterator
    yield futures.pop().result()
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 451, in result
    self._condition.wait(timeout)
  File "/usr/lib/python3.11/threading.py", line 327, in wait
    waiter.acquire()
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mloubout/DevitoCodes/devitopro/mfes/decoupler.py", line 16, in <module>
    main()
  File "/home/mloubout/DevitoCodes/devitopro/mfes/decoupler.py", line 11, in main
    with MPIPoolExecutor(max_workers=4) as executor:
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 647, in __exit__
    self.shutdown(wait=True)
  File "/home/mloubout/.local/lib/python3.11/site-packages/mpi4py/futures/pool.py", line 207, in shutdown
    pool.join()
  File "/home/mloubout/.local/lib/python3.11/site-packages/mpi4py/futures/_lib.py", line 160, in join
    self.thread.join()
  File "/usr/lib/python3.11/threading.py", line 1119, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.11/threading.py", line 1139, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

[dalcinl]

Wow... At this point I'm running out of ideas. What exact Ubuntu release are you using? Are you using Open MPI installed via apt? I want to try reproducing the failure running in a Docker container, but for that I need to know the exact steps to reproduce the environment.

[mloubout]

What exact Ubuntu release are you using?

23.10

Are you using Open MPI installed via apt?

No I'm using openmpi v4.1.4 compiled from source

I want to try reproducing the failure running in a Docker container,

I will work on making a container that reproduces it so you can run it

[dalcinl]

I will work on making a container that reproduces it so you can run it

Or maybe just a bash script that set things up (apt update && apt install ...) such that I can run it locally.

[mloubout]

So somehow (after destroying my system and reinstalling ubuntu) it now works fine.

Will leave this discussion open as we do have some slight issues when using intel mpi and the openmpi version that comes with the nvidia hpc SDK but we'll need to do some more investigating first.

Thank you for your time.

[FabioLuporini]

don't know about Intel, but the issue I saw with the Nvidia HPC SDK is definitely in their MPI, not mpi4py :)