avoid communicating edge and corner boundary data?

[BenWibking]

Hi all,

My Quokka code uses a stencil that is a tensor product of 1-d stencils, so in principle, only the boxes sharing faces need to exchange boundary data. For uniform boxes of grid_size=128 and 4 ghost cells, this leads to most MPI messages (~77 percent) communicating edge and corner data (only ~6 per cent of the boundary data), which is unused by my code. Is there an straightforward way to modify AMReX to avoid the edge/corner boundary communication during the level advance?

Thanks,
Ben

[asalmgren]

Hey Ben, There's a "cross" argument for FIllBoundary that defaults to false -- I think this would work for your question. If you set "cross" to true then you are saying that you are effectively only doing "cross stencils", i.e. those which are only 1d in each direction, i.e. no cross terms, so FillBoundary won't fill those extra ghost cells. From AMReX_FabArray.H: /** * \brief Copy on intersection within a FabArray. Data is copied from * valid regions to intersecting regions of definition. The * purpose is to fill in the boundary regions of each FAB in * the FabArray. If cross=true, corner cells are not filled. * If the length of periodic is provided, periodic boundaries are * also filled. Note that FabArray itself does not contains * any periodicity information. * FillBoundary expects that its cell-centered version of its BoxArray * is non-overlapping. */ void FillBoundary (bool cross = false); void FillBoundary (const Periodicity& period, bool cross = false); void FillBoundary (const IntVect& nghost, const Periodicity& period, bool cross = false); //! Same as FillBoundary(), but only copies ncomp components starting at scomp. void FillBoundary (int scomp, int ncomp, bool cross = false); void FillBoundary (int scomp, int ncomp, const Periodicity& period, bool cross = false); void FillBoundary (int scomp, int ncomp, const IntVect& nghost, const Periodicity& period, bool cross = false);

…

On Wed, Nov 24, 2021 at 10:04 PM Ben Wibking ***@***.***> wrote: Hi all, My Quokka code uses a stencil that is a tensor product of 1-d stencils, so in principle, only the boxes sharing faces need to exchange boundary data. For uniform boxes of grid_size=128 and 4 ghost cells, this leads to most MPI messages (~77 percent) communicating edge and corner data, which is unused by my code. Is there an straightforward way to modify AMReX to avoid the edge/corner boundary communication during the level advance? Thanks, Ben — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2499>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRE6YXXWZSBVUEG6HKXJO3UNXGV3ANCNFSM5IXWGESQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- Ann Almgren Senior Scientist; CCSE Group Lead Pronouns: she/her/hers

[BenWibking]

Yes, that looks like exactly what I am looking for. For multi-level, is it safe to change FillPatchSingleLevel() in AMReX_FillPatchUtil_I.H so that cross=true for the FillBoundary() calls?

[BenWibking]

For debugging purposes, it would be useful to have a MultiFab::contains_nan() that ignores the corner cells. I didn't see a version of that with a cross argument in the doxygen docs. Is there another function that can be used for this?

[WeiqunZhang]

Yes, that looks like exactly what I am looking for. For multi-level, is it safe to change FillPatchSingleLevel() in AMReX_FillPatchUtil_I.H so that cross=true for the FillBoundary() calls?

Yes, I think it's safe. However, that does not avoid all the corners. We could add an option for cross type to FillPatch functions.

For debugging purposes, it would be useful to have a MultiFab::contains_nan() that ignores the corner cells. I didn't see a version of that with a cross argument in the doxygen docs. Is there another function that can be used for this?

We could add an variant for it.

[BenWibking]

Those would both be useful to have.