CPU usage pegged at 100%

[abacaj]

I start 4 processes, main process 0% usage but I see 100% CPU usage on the other processes when waiting for a message:

mpirun -n 4 --allow-run-as-root python mpi_test.py

from mpi4py import MPI
import time
import os


def mpi_comm():
    return MPI.COMM_WORLD


if __name__ == "__main__":
    if os.environ["OMPI_COMM_WORLD_NODE_RANK"] == "0":
        time.sleep(10)
        mpi_comm().isend(["hello"], dest=1)
        mpi_comm().isend(["hello"], dest=2)
        mpi_comm().isend(["hello"], dest=3)
    else:
        req = mpi_comm().irecv(source=0)
        data = req.wait()
        print(data)

Is this expected (couldn't really find a answer elsewhere):

[dalcinl]

If you do a quick Google search for let say "MPI CPU usage 100%", you may see dozens of very similar reports. This is the behavior of most MPI implementations: they spin the CPU busy-waiting for the arrival of new messages. This behavior has nothing to do with mpi4py, nor mpi4py has a way to directly act on it. You should read the documentation of your backend MPI implementation or ask its support/development team for ways to alleviate the issue, if any.

Alternatively, you can change your code and do things slightly different. Instead of a blocking wait(), you can use test() in a while True: ... loop and break when the message arrives or otherwise time.sleep(delta). Using delta=1e-4 maybe a good compromise between the added communication latency and CPU spinning, but that ultimately depends on the requirements of your actual application. A slightly better approach than a small constant delta is to use exponential backoff, that is what mpi4py.futures uses to alleviate the high-CPU issue (I don't want mpi4py user's laptops to burn), check the source code, you should get the idea.

[abacaj]

Thanks for the quick reply, this is helpful.

[dalcinl]

Let's keep this discussion open as a reference for others that may experience the same problem.

[abacaj]

Turns out one can also set mpi_yield_when_idle to move to degraded busy wait.

mpirun --mca mpi_yield_when_idle 1 -n 4 --allow-run-as-root python mpi_test.py

without mpi_yield:

with mpi_yield:

[dalcinl]

FWIW, I've just added this settings to the mpi4py/setup-mpi action. Thanks for the tip!