Questions concerning MPI.Win

[DirkKuhn]

Hi,

I have a couple of questions concering the MPI.Win class. I hope you can help me.

What is the recommended way to access the memory in a process's own window?

Consider for example that a master process (rank=0) may at some time set a flag in the worker process (rank=1) window

true_value = np.ones(1, dtype=np_dtype)
win.Lock(rank=1)
win.Put(true_value, target_rank=1, target=(pos, 1, datatype))
win.Unlock(rank=1)

Between computations the worker process (rank=1) checks whether the falg is set. Which of the following alternatives is preferred?

buf = np.zeros(N, dtype=np_dtype)
while ...:
    # do some computation
    win.Lock(rank=1)
    win.Get(buf, target_rank=1)
    win.Unlock(rank=1)
    # check buf

or

buf = memoryview(win)
while ...:
    # do some computation
    # check buf[pos]

or

buf = np.frombuffer(win, dtype=np_dtype)
while ...:
    # do some computation
    # check buf[pos]

In the last two cases should there be win.Lock and win.Unlock called before and after access to buf?

[DirkKuhn]

Let's say the master process should set the flag in multiple worker processes. Is there a broadcast operation or is the following necessary?

true_value = np.ones(1, dtype=np_dtype)
for worker_rank in range(1, size):
    win.Lock(rank=worker_rank)
    win.Put(true_value, target_rank=worker_rank, target=(pos, 1, datatype))
    win.Unlock(rank=worker_rank)

[dalcinl]

No, there is no broadcast operation. Maybe you could use Lock_all/Unlock_all with the loop in between rather than locking/unlocking each rank.

[jeffhammond]

if it is collective, having each receiver of the broadcast call Get is better than the root calling Put N times.

[DirkKuhn]

Thank you for your answers! Is calling Get still better if Get would be called hundereds of times more often than Put? Considering that Get called in the worker would have to access the window in the root process.

[DirkKuhn]

I don't think that MPI.Win.Allocate ensures initialization with zeros. What is the recommended way to fill each process's buffer with zeros?

buf = np.zeros(N, dtype=np_dtype)
win.Lock(rank=rank)
win.Put(buf, target_rank=rank)
win.Unlock(rank=rank)

or just

buf = np.frombuffer(win, dtype=np_dtype)
buf.fill(0)

Should win.Lock and win.Unlock be called in the second case?
Thank you very much!

[dalcinl]

I don't think that MPI.Win.Allocate ensures initialization with zeros.

You are right, it does not.

What is the recommended way to fill each process's buffer with zeros?

I would do it this way:

win = MPI.Win.Allocate(...)
np.frombuffer(win, dtype='B').fill(0)
win.Fence()

or perhaps

win = MPI.Win.Allocate(...)
win.tomemory()[:] = 0
win.Fence()

EDIT: Just in case, I added Fence() calls after zeroing.

[jeffhammond]

yeah or use MPI_Win_sync if you don't need the barrier semantic of MPI_Win_fence.

[dalcinl]

@DirkKuhn Your questions are not really about mpi4py but way more general, about how MPI behaves. Unfortunately, I'm not a guru of MPI RMA, and the whole thing is still sometimes confusing to me. Maybe @jeffhammond can provide us some links or resources to better understand things? Sorry, Jeff, I always end up involving you in mpi4py RMA questions.

[dalcinl]

Regarding you opening questions, I believe the second and third cases (using memoryview and np.frombuffer respectively) still need the lock/unlock bracket. I believe using Get() to self-rank vs a direct load/stores via buffer is equivalent, but I may very well be wrong.

[jeffhammond]

Get and Put from/to self is going to be equivalent to memcpy. Depending on the synchronization mode, there may be a lock involved in some cases, although perhaps only when using accumulate functions, which are atomic.

[jeffhammond]

My recommendation for RMA usage is:

1. Construct and lock a window

// always do these together
{
MPI_Win_allocate
MPI_Win_lock_all
}

2. Synchronize

// three types of sync
MPI_Win_flush_local // buffer can be reused - like MPI_Send
MPI_Win_flush       // operation is completed at the target - stronger than MPI_Ssend
MPI_Win_sync        // think memory barrier - before and after load-store access to local window

3. Destroy window

// always do these together
{
MPI_Win_unlock_all
MPI_Win_free
}

[DirkKuhn]

Do I understand it correctly that after MPI_Win_allocate is called all windows stay locked until they are freed? If so, would that not be a problem with exclusive locks (I assume I misunderstand what a lock is in this context). What is the benefit of this compared to just locking the Put and Get calls?
I am sorry for my poor understanding of MPI.

[jeffhammond]

Locking on every call has overhead. If shared lock is the only usage, hoist it out. Exclusive lock isn't very useful. Accumulate is elementwise atomic. That's often good enough. Build proper locks with CAS if necessary. Win lock is a transactional semantic that doesn't match most parallel patterns.

[jeffhammond]

anything you can do with Fence you can do with the above. PSCW is harder to reproduce with passive target, but it's possible. i just don't recommend it.