[WIP] EBs: Compiled by Default, Controlled at Runtime

[ax3l]

Enable embedded boundary compilation by default.
Control via runtime option (default: off).

Close #4863

Action Items

• build system & doc default updates
• introduce runtime control, update runtime logic
• run a non-EB test, ensure it is as fast and as low high-water mark in memory consumption as with EB compile-time disabled

[dpgrote]

This is a good thing to do. Though, is a separate input flag needed? The code can check if warpx.eb_implicit_function has been specified, or eb2.geom_type. Elsewhere, enable type input parameters have been removed, relying on specification of the controlling parameters.

[ax3l]

Thanks, Dave!

I was wondering as well, but in the end we have at least three ways to turn it on:
https://warpx.readthedocs.io/en/latest/usage/parameters.html#embedded-boundary-conditions

• warpx.eb_implicit_function
• warpx.eb_potential(x,y,z,t)
• and, currently undocumented, a way to load via STL file in AMReX

We could a) fix the lack of docs and b) write a helper function that checks them or so? Open to suggestions cc @RemiLehe @roelof-groenewald

[roelof-groenewald]

The warpx.eb_potential(x,y,z,t) parameter would not be good to use since it is perfectly fine to leave that out - it is only used by the electrostatic solver and by default the embedded boundary potential is just 0.
I think it would be reasonable to then check if either warpx.eb_implicit_function or the STL file input is specified. We don't expect to have a lot more ways of loading EBs in the future, right? So it shouldn't be hard to maintain a list of or statements to determine the runtime parameter.

[roelof-groenewald]

It would be nice if we can always have a well defined EBFactory which just doesn't cover any of the domain when it is not wanted. Then the edge_lengths and similar multifabs could be properly defined in all cases and we wouldn't have to have the if (eb_enabled) block. Similarly lime 131 below could simply be:

if (lr(i, j, 0) <= 0) return;

and the same for all the other similar occurrences.

[WeiqunZhang]

That would waste memory.

[ax3l]

Unless we would implement an Array4 variant that returns 0 outside of the covered domain, which we could use elsewhere as well (e.g. #4807). But that then probably leads to register bloat (unless we implement as in mdspan a variant that have optionally known compile-time sizes).

[WeiqunZhang]

A separate bool ParmParse parameter is not needed unless we want to the one flag to rule them all. If we want a bool variable to be used in the code for convenience, we can add a variable to WarpX class. EB is initialized in WarpX::InitEB. There we can set it to true or false.

There are basically two ways to enable EB in WarpXInitEB, (1) warpx.eb_implicit_function (2) AMReX's eb2.geom_type. The STL way is the second way documented here https://amrex-codes.github.io/amrex/docs_html/EB.html?highlight=stl.

[ax3l]

Thank you for all the ideas!

we can add a variable to WarpX class. EB is initialized in WarpX::InitEB. There we can set it to true or false.

In ImpactX and also more in WarpX, I try to avoid using global variables in the central sim class and instead use globals from the ParmParser. That way, we do not need to keep them in sync when changing options at runtime (e.g., from Python) and do not need to include a heavy central class everywhere.
But I also just introduced WarpX::m_eb_enabled... 😅

I would for now write a free standing helper function instead that we can use :)

[WeiqunZhang]

Maybe a free function in WarpX like this

bool hasEB ()
{
#if defined(AMREX_USE_EB)
    return ! amrex::EB2::IndexSpace::empty();
#else
    return false;
#endif
}

[WeiqunZhang]

Note that amrex::IndexSpace has a static member, a vector of amrex::IndexSpace. Somehow this comes into my mind by your mention of global variable. https://en.wiktionary.org/wiki/%E6%88%91%E4%B8%8D%E5%85%A5%E5%9C%B0%E7%8D%84%EF%BC%8C%E8%AA%B0%E5%85%A5%E5%9C%B0%E7%8D%84

[WeiqunZhang]

Note that amrex::EB2::IndexSpace::empty() is true before WarpX::InitEB is called and will be false if EB is initialized in that function.