MPI Related Question

[alahyari]

Hello guys,

In a parallel application, I need to chop of the main MultiFab solution into smaller pieces and calculate a certain variable in each piece. Let's say I have a single level (32x32x32) grid, and I chop it off to (8x8x8) boxes, and calculate some variable and store it in a new MultiFab built on a (4x4x4) box and the same dm as the original grid. This is all done in parallel, but at the end of the calculation I need to make this new MultiFab available to all processors for some local calculation. Is there a built-in feature in AMReX to do so?

Thank you.

[WeiqunZhang]

but at the end of the calculation I need to make this new MultiFab available to all processors for some local calculation.

Could you clarify what that means? Do you mean each process needs access to all the data including those owned by other processes?

[alahyari]

That is correct. In theory and for this specific application, it shouldn't be an excessively large array (between 8^3 to 16^3 cells with 6 components).

[WeiqunZhang]

MultiFab is a distributed data container. If you need all processes to have a copy of the entire data, you can store the data in a 1D vector and use MPI_Allgather to gather the data. You can use FArrayBox (const Box& b, int ncomp, Real* p) noexcept; to build multiple non-owning Fabs so that you can still view the 1D data as a multi-d array.

[alahyari]

Thank you @WeiqunZhang. This was very helpful.