Internal storage for MPIPoolExecutor #370
jan-janssen
started this conversation in
Ideas
Replies: 1 comment
-
In Within the initializer function, you can store your data in a global variable using the from mpi4py.futures import MPIPoolExecutor
data = None
def init(arg):
import __worker__
__worker__.data = arg
def task(arg):
return data + arg
if __name__ == '__main__':
with MPIPoolExecutor(initializer=init, initargs=(40,)) as e:
print(e.submit(task, 2).result()) $ mpiexec -n 1 python script.py
42 |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I would like to store data or even a python object inside the
MPIPoolExecutor
and then submit functions to interact with this object. For example:Here
k
is used from the internal storage so the output is10
. This code example is based on:https://pympipool.readthedocs.io/en/latest/interfaces.html#executor
Which I currently use to identify features I would like to have available in the general
MPIPoolExecutor
. While in this example the overhead of communicatingk
multiple times is minimal,k
could also reference a several GB large array I would like to analyze. So keeping it in memory can be highly beneficial.Beta Was this translation helpful? Give feedback.
All reactions