Adaptive mesh refinement API

[termi-official]

In #780 Maxi develops the main infrastructure for "p4est-type" AMR~1 (which we currently name BGW internally, derived from the author names). In addition I am planning to implement "multilevel-type" AMR [2]. This raises the question how an API might look like for using these meshes efficiently and which key components can be reused. Specifically

1. Can we work directly on the mesh representations and get away with a single dof handler by being clever with the dispatches?
2. How does the interaction with constraint handler look like?
3. How can we efficiently reuse our FEValues? I.e. reinit for multilevel type meshes work similar as reinit for Nedelec elements (i.e. need a pre-processing step of the basis functions)
4. How to transport solution vectors between consecutive AMR steps?
5. Whch components can be reused between AMR implementations (e.g. balancing?)
6. How to iterate and integrate over faces (see InterfaceValues for DG interface integration #743 (comment))

References

1. Burstedde, Carsten, Lucas C. Wilcox, and Omar Ghattas. "p4est: Scalable algorithms for parallel adaptive mesh refinement on forests of octrees." SIAM Journal on Scientific Computing 33.3 (2011): 1103-1133.
2. Zander, Nils, et al. "Multi-level hp-adaptivity: high-order mesh adaptivity without the difficulties of constraining hanging nodes." Computational Mechanics 55.3 (2015): 499-517.

[koehlerson]

After some off time, Dennis pushed #780 further to include rotated cells and therefore unstructured grids. I think 2D works, so it's somehow time to think about the listed things. Especially the first two need somehow to be cleared.

If we proceed with a single DofHandler then we need to change things like https://github.com/Ferrite-FEM/Ferrite.jl/blob/master/src/Dofs/DofHandler.jl#L526 or at least change dispatches such that in cases, where we don't have the ownership of entities belonging to the Cell datastructure (as in the p4est implementation or distributed scenario). Further, we would probably need some sort of Union{Nothing,AdaptivityBuffer} as field in the DofHandler to have some flexibility in terms of storing mappings, hanging nodes and the like.

I'm very unsure about the ConstraintHandler part since the last time I touched something with constraints is quite long ago. In my ideal case we would have the construction/update of the ConstraintHandler that automatically applies the affine constraints for the hanging nodes

[kimauth]

Really cool work you're doing here!

Regarding the DofHandler, I would advocate for starting with a separate implementation for the beginning, even if that means significant code duplication to start with. Once the implementation settles we can still figure out if joining the two implementations would be beneficial (similar to how the MixedDofHandler was a separate thing for a long time). Major reasons:

• It allows you to be more flexible in trying out designs without being restricted by the current design.
• All of us other possibly working on dof distribution don't need to learn all internals for p4est in order to do any development at all (and won't break things for p4est).

I love all of the new feature that Ferrite got recently, but I've also gone through quite a lot of frustration about having to learn about very involved (and perhaps not widely used) features, whenever I was trying to do some work on Ferrite again. Thus I want to campaign a little for implementing "fancy" features separated from core functionality as far as possible. 🙂

[termi-official]

Really cool work you're doing here!

Thanks! Also to Maxi for the heavy lifting here.

Regarding the DofHandler, I would advocate for starting with a separate implementation for the beginning, even if that means significant code duplication to start with. Once the implementation settles we can still figure out if joining the two implementations would be beneficial (similar to how the MixedDofHandler was a separate thing for a long time). Major reasons:

* It allows you to be more flexible in trying out designs without being restricted by the current design.

* All of us other possibly working on dof distribution don't need to learn all internals for p4est in order to do any development at all (and won't break things for p4est).

I want to throw in the idea that think we might be able to also get away with more dispatches in close!.

I love all of the new feature that Ferrite got recently, but I've also gone through quite a lot of frustration about having to learn about very involved (and perhaps not widely used) features, whenever I was trying to do some work on Ferrite again. Thus I want to campaign a little for implementing "fancy" features separated from core functionality as far as possible. 🙂

Can we open a separate issue to discuss this (also for visibility) and could you point therein towards the relevant portions? This might be something we should adjust before 1.0 release in one way or another.

[fredrikekre]

A separate dofhandler is probably the easieast. It can still probably reuse lots of existing utility functions and such. You still need to update the grid type for doing adaptive analyses so updating the dofhandler type too is not really a big deal.

[koehlerson]

#780 is now able to refine, coarsen, balance and materialize a mesh in 2 and 3D (for rotated initial meshes as well). Before I start writing the docs, I'd like to ask here what the opinions are about either splitting it into a separate package vs. keeping it in Ferrite's core.

From my side, I am afraid that infrastructure that requires deep interaction with other components of Ferrite's core might be split into multiple packages and people might miss composability when writing code in the separate packages. For example, if GPU and distributed support lands one day, it would be great to extend the p4est stuff directly with it by being consistent with some interface. Of course, it is possible to make this scenario work as well, but if we go that way, then we need some cross-package testing so that we test for cases that should be composeable. Another point regarding splitting into different packages is that I expect that developing features that require changes to Ferrite's core can become tedious. For example, in the adaptive case, if we throw away the node-dof mapping dicts in the DofHandler, it's annoying to handle that in a separate package and wait for changes in the PR, version constraints, etc. If the feature, in this case adaptivity, is shipped directly with the core, there is only one matching version.

On the other hand, I can see the point that it is cumbersome to consider all the needs of all the included features, especially when everything is packaged in a single repo.

[KnutAM]

First of all, thanks for all the great effort on developing 780 🚀
No conclusions, just sharing my initial thoughts in case relevant, please ignore if not :)

I'm split regarding if splitting into more packages will make it easier or not.
On one hand, @fredrikekre convinced me that via cross-package testing (in the Ferrite.jl test suite) will give a warning if you break something, and you can choose if this was intentional (i.e. something internal was used and leave to the other package's devs to fix) or intentional and don't break it. But I also fear that if other packages rely on a lot of Ferrite internals, it will become difficult to avoid breaking them when changing Ferrite, which could prevent the Ferrite core development / require some strange workaround hacks (or synced changes)

people might miss composability when writing code in the separate packages

I would have thought that splitting into packages would rather encourage composability to a larger extent by defining and using utilities and abstract interfaces more consistently. Both the DofHandler and ConstraintHandler could probably benefit from this?

For example, in the adaptive case, if we throw away the node-dof mapping dicts in the DofHandler, it's annoying to handle that in a separate package and wait for changes in the PR, version constraints, etc. If the feature, in this case adaptivity

While I see this issue, I guess the advantage would be that changes to a separate package might be easier to get in, and thus new features are available faster. Only in the case it requires changes to Ferrite core it would take longer, but that would take longer with every option I think?

Lastly, regarding defining and using utilities and abstract interfaces more consistently: For the DofHandler example, I would expect that generalizing the dof-distribution to be dof-handler independent would be the solution, and then it would be super-easy to add a custom DofHandler. But, I fear that making a lot of these abstractions could also make it even more challenging to understand Ferrite core.

[koehlerson]

But I also fear that if other packages rely on a lot of Ferrite internals, it will become difficult to avoid breaking them when changing Ferrite, which could prevent the Ferrite core development / require some strange workaround hacks (or synced changes)

When that happens, you still have to figure out what went wrong and communicate with the people who developed some other feature that relied on the core functionality. The benefit of splitting it into a package disappears somewhat for me there, in terms of arguing for easier maintenance of the core. But maybe I am missing something.

I would have thought that splitting into packages would rather encourage composability to a larger extent by defining and using utilities and abstract interfaces more consistently. Both the DofHandler and ConstraintHandler could probably benefit from this?

Somehow, I think. The problem is that some interfaces won't work with more advanced features, and I'm not sure it's beneficial to have the more advanced features in different packages, because I see that some argument might start that this interface change (which may or may not be a substantial change with more complex code) is only required by some subpackage. Thus, the interface will not be functional for the feature being developed.

While I see this issue, I guess the advantage would be that changes to a separate package might be easier to get in, and thus new features are available faster. Only in the case it requires changes to Ferrite core it would take longer, but that would take longer with every option I think?

Won't be fast to get the changes in, true. But there is more than time. You'd need a Ferrite#master CI to test the changes, which may or may not already contain other breaking changes. At the same time, you want it to work with the current release, so this is basically the extra hassle I meant by this example. Not saying that it is impossible, just saying that it introduces some additional maintenance at a different spot.

[KnutAM]

But I also fear that if other packages rely on a lot of Ferrite internals, it will become difficult to avoid breaking them when changing Ferrite, which could prevent the Ferrite core development / require some strange workaround hacks (or synced changes)

When that happens, you still have to figure out what went wrong and communicate with the people who developed some other feature that relied on the core functionality. The benefit of splitting it into a package disappears somewhat for me there, in terms of arguing for easier maintenance of the core. But maybe I am missing something.

Yes, I agree, this was an argument against splitting into separate packages.

[koehlerson]

gentle ping 👉 👈

[KristofferC]

Just a small opinion from my side, I think premature splitting out into other packages is much worse than some small bloat of Ferrite itself.

[koehlerson]

I discussed a bit with @fredrikekre and I think the result is now to have a submodule Ferrite.AMR which is part of the repo containing all things related to the AMR implementation.