Example use case: FSI in helicopters

I will probably implement this eventually.

I am currently encountering this in my work on TherMoS, too: Here, coordinates of the sun and a transform are exchanged between the solvers.

Is there already any progress on this issue? Otherwise I am going to work around this by introducing an auxiliary mesh.

@uekerman Really good post. Just what I was looking for. (I know WIP).
Commenting on this to let you guys know there is demand for this (but no hurries).:stuck_out_tongue_winking_eye:

I have been trying to make my volume coupling simulations faster by trying to transfer physical constants as scalars or single point mesh instead of another datafield with all the mesh points. But your idea is better as we couple them once instead of in each coupling time step.
Once this is implemented would be perfect to transfer physical constants such as density, viscocity, conductivity, specific heat etc.

But your idea is better as we couple them once instead of in each coupling time step.

To avoid misunderstandings: this feature would still communicate the global data in every advance. Is this would you are looking for or do you additionally need a feature to communicate certain data only during initialization (if so, we should open an additional issue for this)?

Is this would you are looking for or do you additionally need a feature to communicate certain data only during initialization (if so, we should open an additional issue for this)?

@uekerman Yes this can be separated into a different issue (maybe an extension of the current issue). As certain physical constants don't change for the duration of the simulation.
I think issue #464 might have misled me.

@Alphaoo1 Please see #1071

This issue has been mentioned on preCICE Forum on Discourse. There might be relevant details there:

https://precice.discourse.group/t/possibility-of-coupling-openfoam-python-solver-without-meshes-and-no-requirement-for-mapping/874/4

Some comments on the feature.

Configuration

Why the name global-data in the configuration?
Global is a strong word and I would expect this to be accessible from everywhere even without to specify read-data or write-data.
I don't have a better suggestion than meshless-data though, which I don't really like.

What about adding an attribute to the data tag instead?

  <data:scalar name="ClassicalMeshData"/>
  <data:scalar name="GlobalData1" global="true"/>
  <data:scalar name="GlobalData1" meshless="true"/>

Or we simply derive the state of being meshless from the config. If no mesh uses a data, then it is meshless. This sound tedious to implement though.

API

Why the globalDataIDs?.

Given that there is no association to vertices, there is no reason for indexing. We tell preCICE what size we expect, hence we also what we can read/write.

// Caller knows the dimensionality of the data.
void setGlobalDataSize (
    int     dataID,
    int     size);

// To avoid problems with data dimensionality, we could also return the amount of elements.
int setGlobalDataSize (
    int     dataID,
    int     size);

// expect size * getDimensions() values
void writeGlobalVectorData(
    int     dataID,
    int     size,
    const double* values );

// expect size * getDimensions() values
void readGlobalVectorData(
    int     dataID,
    int     size,
    double* values );

We could optionally allow an optional write/read with offset to allow extracting specific entries:

void readGlobalVectorData(
    int     dataID,
    int     size,
    int     offset,
    double* values );

What about other data types?

Currently, everything has to be encoded in double. Isn't this data way more prone to be used as state flags or similar?

Communication behavior

Writing should be allowed on any rank. If data is written on multiple ranks, preCICE checks for consistency.

Why this additional complexity? Isn't it easier to require the Primary rank to write?
If we require consistency, then partial writes are not supported.

Alternative would be to sum everything up on the primary rank, this way we could support partial writes at some additional overhead.

Developer perspective

Doesn't #1350 simplify this a lot?

We could require global data to be written before initialize(). Then broadcast the size to the other participants based on the exchange tag.

I like the meshless term! 👍 It is much clearer than global. Some other term could be nicer, but this already feels good enough.

I like the meshless term!

Me not 😁 🙈
meshless stands pretty much for meshless / meshfree methods, so still space-dependent data.

Global is a strong word and I would expect this to be accessible from everywhere

In a way, this is what I had in mind. Accessible from everywhere, meaning from every rank, from every location. But maybe we find a better name, sth like space-independent?

A few other good points, will comment later.

I'll be working on this issue :)

Good points. There are indeed still a few open aspects.

What about adding an attribute to the data tag instead?

Yes, possible. Maybe depends on whether we add more attributes (see size below).

Or we simply derive the state of being meshless from the config. If no mesh uses a data, then it is meshless. This sound tedious to implement though.

I agree

Why the globalDataIDs?.
Given that there is no association to vertices, there is no reason for indexing. We tell preCICE what size we expect, hence we also what we can read/write.

The reasoning for globalDataIDs is the same as offset. But maybe indeed this is not needed.

Maybe everything gets easier if we drop the size altogether? Not sure yet. What I mainly had in mind was really to communicate material or geometry parameters. And if there are two parameters, one could create two data fields.

If we keep the size we have the additional complexity of which participant defines the size. Maybe directly doing it in the configuration is simpler:

<data:scalar name="GlobalData1" global="true" size="5"/>

size could default to 1.

Seems like I didn't see this difficulty when opening the issue 4 years ago 😁

What about other data types?
Currently, everything has to be encoded in double. Isn't this data way more prone to be used as state flags or similar?

Could be true, yes, but I would argue that it is misused then. IMO it should not be used for metadata, but for physical values, same as our current data fields.

Writing should be allowed on any rank. If data is written on multiple ranks, preCICE checks for consistency.
Why this additional complexity? Isn't it easier to require the Primary rank to write?
If we require consistency, then partial writes are not supported.
Alternative would be to sum everything up on the primary rank, this way we could support partial writes at some additional overhead.

In theory, the user does not know about the primary rank. I don't see a use case for partial writes right now, is there one? I could imagine, however, that a user simply writes a material parameter on every rank for simplicity.

Doesn't #1350 simplify this a lot?

Yes, possibly. I would also suggest to start development by branching of from develop-v3.0.0.

We could require global data to be written before initialize().

This I don't understand. I would still communicate data always, not only during initialization (as suggested in #1071). Imagine an angle between coordinate systems that changes in every timestep / iteration.

FYI: While discussing with @thesamriel on how to perfectly couple domains with conforming but non-orthogonal meshes, I was thinking that this feature could be useful to also exchange some parameters of the mesh. Just documenting for now, not sure how relevant this can be in practice.

We could require global data to be written before initialize().

This I don't understand. I would still communicate data always, not only during initialization (as suggested in #1071). Imagine an angle between coordinate systems that changes in every timestep / iteration.

@uekerman If we know the data size before initialize, then we can deduce the common size and/or make sure that the size is consistent everywhere.
The problem is still that we need a system to deduce the size.

Other idea:

The writing participant writes data locally on each rank (or not).
This is then concatenated on the primary rank, preserving order among ranks. The result is then sent to the receiving participants and scattered across all rank, which can access the entire thing.

This would also play nicely with the partitioning philosophy of preCICE, as the receiving global data has a consistent shape, independent on how the "writing side" of the data is partitioned.

Maybe some user experience is useful here.

From my experience with CAMRAD-TAU coupling, and what I see from other multibody dynamics cases.

CSM side only has one rank, i.e. no worries about writing/reading.

CFD side the problem gets a bit complicated but not so much.

All ranks should be able to access all data.

All ranks at the CFD side should be able to write all vector data. action:summation should be able to apply to the resultant vector.

Use case example

CAMRAD has to send collocation points to TAU -> required one time at the beginning of coupling a vector data of size x
TAU has to send collocation moments/forces to CAMRAD -> required at each time step a vector data of size x

Since CAMRAD is a single executable there is nothing to say about it.

TAU is MPI parallel, some nodes have the surface some are not ie. not all of them have to send data.

If we consider 1 collocation point and let's say rank 1 has the upper surface of a blade and rank 2 has the lower surface of the blade.

The resultant force is coming from sum(vec_rank1 + vec_rank2) I have never used the action:summation but it should be able to handle the reduction of the data vector before reading right?

This also super simplifies CAMRAD-TAU coupling or any future Multibody CSM - CFD coupling. Actually, they don't require any mapping future of preCICE.

I am in favor of setting the size in the preCICE config rather than handle with API, if we add a setter then we also need a getter on the read-side.

// Return the number of elements
int getGlobalDataSize (int  dataID)

@kursatyurt Thanks for the input!

I am not sure if collocation points are the best example here. The data should be space-independent. What you describes sounds a lot like https://precice.org/couple-your-code-direct-access.html, would you agree?

@kursatyurt Thanks for the input!

I am not sure if collocation points are the best example here. The data should be space-independent. What you describe sounds a lot like https://precice.org/couple-your-code-direct-access.html, would you agree?

Yes, this sounds better, I was not aware of this feature. Probably at the time when the adapter code was written these features were not there. Of course, there are tons of space-independent data like rotor radius, rotational speed, the number of Fourier harmonics used, etc.

This still does not cover the case per vertex we might have n-dimensional data. Let's discuss this at another issue.

Note: This comment may be discussed in #1525

Not directly related to this issue, but came across this while working on it.

In the data read/write functions, we have

PRECICE_CHECK(valueIndex >= -1,

followed a few lines later by

PRECICE_CHECK(0 <= valueIndex && valueIndex < vertexCount,

And valueIndex doesn't change anywhere in these functions.

Then isn't the first check redundant?

Regarding DataContext:

DataContext's constructor currently has a PRECICE_ASSERT(mesh); so we can't simply use it for meshless/global data.

Ideas:

• Overload the constructor to suit global/meshless datas.
• Inherit GlobalDataContext and MeshDataContext from DataContext. Remove PRECICE_ASSERT(mesh); from DataContext and put it in MeshDataContext.

I personally prefer the first one.

Regarding Data:

Data class is currently in the precice::mesh namespace.
I suggest it should not be under meshnow that it is being used for global/meshless data as well. Thoughts?

Edit:

Just saw that the original post suggests a GlobalData class. GlobalData would be almost identical to Data, just without the gradient information. So two independent classes for Global and Mesh Data would have repeated code. Moreover, a new GlobalDataclass means DataContext would also need to be specialized to handle GlobalData.
One idea around this is - have a parent class Data, and inherit MeshData and GlobalData as child classes. Then DataContext can simply interface with the parent class.

Another idea is, keep using Dataclass for both Mesh and Global Data; make changes in DataContext instead (the two points described under "Regarding DataContext")

Note: This comment may be discussed in #1525

Yes, let's discuss there

Regarding DataContext

DataContext and a potential GlobalDataContext might not even have so much in common:

• GlobalDataContext has no mesh
• GlobalDataContext has no mapping
• I currently don't think, we need to distinguish between read and write for GlobalDataContext

The only thing both might have in common is the waveform treatment. Sth that is also still undergoing heavy changes right now, cf. #1523.

I currently don't see a need to inherent from a common base class. Both will always be collected in different data structure in SolverInterfaceImpl anyway.

Regarding Data

Data class is currently in the precice::mesh namespace.
I suggest it should not be under mesh now that it is being used for global/meshless data as well. Thoughts?

Alternative could be to rename the mesh namespace as it always was a namespace for meshes and data anyway.

So two independent classes for Global and Mesh Data would have repeated code.

That's no big concern IMO. A GlobalData class will be minimal. Refactoring here should also be easy later on.

Moreover, a new GlobalData class means DataContext would also need to be specialized to handle GlobalData.

No issue if we have a separate GlobalDataContext, see above.

A new contender for the name: "arbitrary" data? (as seen in Y. Fischler's presentation in Workshop 2023)

* How do we treat global data in `CouplingData`? New similar data structure? This has consequences in `_sendData` in `BaseCouplingScheme`

Creating a new GlobalCouplingData for now makes sense to me. Similarly a _sendGlobalData (and GlobalDataMapand new associated functions.)

An open point for discussion: for implicit coupling, what convergence and acceleration would mean in the context of global data.

I think we should also allow using global data in convergence measures and acceleration. Maybe an open problem: does the preconditioner in the acceleration work sufficiently well for global data? But this could also be a follow-up issue.

Update on this issue. In principle, the functionality is ready with #1549, however, this would introduce a lot of technical debt in areas that are currently very heavily on by @BenjaminRodenberg and me. I had an extensive look at simplifying the feature, but it would break some assumptions which lead to a lot of code duplication.

Given that the current work-around using a pseudo-mesh is trivial, we decided to explain it in the documentation and put a halt on the original contribution until we figure out how to progress.