Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mpiexec hangs with multiprocessing with a non-fork start method #501

Open
mihaic opened this issue May 24, 2021 · 0 comments
Open

mpiexec hangs with multiprocessing with a non-fork start method #501

mihaic opened this issue May 24, 2021 · 0 comments
Labels

Comments

@mihaic
Copy link
Member

mihaic commented May 24, 2021

See the following example:

mpiexec -n 2 python3 test_pool.py  # runs forever, kill it manually
cat test_pool.py
import multiprocessing

from mpi4py import MPI

def test_pool():
    multiprocessing.set_start_method("spawn")
    with multiprocessing.Pool(2) as pool:
        pool.apply(f, ())

def f():
    pass

if __name__ == "__main__":
    test_pool()

This affects us when running on macOS, where the the start method is "spawn". I will disable the affected tests so CI can pass.

We should probably disable multiprocessing in this situation.

@mihaic mihaic added the bug label May 24, 2021
mihaic added a commit to mihaic/brainiak that referenced this issue May 24, 2021
Fixes issue brainiak#499.

Also made a number of other changes in the process of getting GitHub
Actions to run. I think it makes sense to include them here as opposed
to making separate PRs for them.

Add workaround for editable user install.

Ignore flake8-docstrings errors (not part of CONTRIBUTING).

Use flake8 extend-ignore instead of explicit ignore.

Add IGNORE_CONDA env variable for pr-check to use Venv even when Conda
is available.

Drop unused Jenkins script.

Remove editable install from requirements-dev.txt to speed up testing.

Fix missing matnormal in pr-check.

Disable tests with mpiexec with non-fork multiprocessing (issue brainiak#501).
mihaic added a commit to mihaic/brainiak that referenced this issue May 24, 2021
Fixes issue brainiak#499.

Also made a number of other changes in the process of getting GitHub
Actions to run. I think it makes sense to include them here as opposed
to making separate PRs for them.

Add workaround for editable user install.

Ignore flake8-docstrings errors (not part of CONTRIBUTING).

Use flake8 extend-ignore instead of explicit ignore.

Add IGNORE_CONDA env variable for pr-check to use Venv even when Conda
is available.

Drop unused Jenkins script.

Remove editable install from requirements-dev.txt to speed up testing.

Fix missing matnormal in pr-check.

Disable tests with mpiexec with non-fork multiprocessing (issue brainiak#501).
mihaic added a commit that referenced this issue May 25, 2021
Fixes issue #499.

Also made a number of other changes in the process of getting GitHub
Actions to run. I think it makes sense to include them here as opposed
to making separate PRs for them.

Add workaround for editable user install.

Ignore flake8-docstrings errors (not part of CONTRIBUTING).

Use flake8 extend-ignore instead of explicit ignore.

Add IGNORE_CONDA env variable for pr-check to use Venv even when Conda
is available.

Drop unused Jenkins script.

Remove editable install from requirements-dev.txt to speed up testing.

Fix missing matnormal in pr-check.

Disable tests with mpiexec with non-fork multiprocessing (issue #501).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant