[WIP] Custom Hierarchies #1432

franzpoeschel · 2023-05-03T12:04:33Z

TODO

Diff: https://github.com/franzpoeschel/openPMD-api/compare/topic-remove-scalar-component..topic-custom-hierarchies

include/openPMD/backend/BaseRecord.hpp

+
+private:
+    template <typename... Arg>
+    iterator makeIterator(Arg &&...arg)


include/openPMD/backend/BaseRecord.hpp

+        return iterator{this, std::forward<Arg>(arg)...};
+    }
+    template <typename... Arg>
+    const_iterator makeIterator(Arg &&...arg) const


test/CoreTest.cpp

+    REQUIRE(r["x"].resetDataset(dset).numAttributes() == 0); /* unitSI */
+    // REQUIRE(r["y"].unitSI() == 1);
+    REQUIRE(r["y"].resetDataset(dset).numAttributes() == 0); /* unitSI */
+    // REQUIRE(r["z"].unitSI() == 1);


test/CoreTest.cpp

+    // unitSI is set upon flushing
+    // REQUIRE(r["x"].unitSI() == 1);
+    REQUIRE(r["x"].resetDataset(dset).numAttributes() == 0); /* unitSI */
+    // REQUIRE(r["y"].unitSI() == 1);


test/CoreTest.cpp

@@ -966,6 +968,27 @@
 #endif
 }

+TEST_CASE("baserecord_test", "[core]")


include/openPMD/backend/BaseRecord.hpp

+    // for (auto it = this->container().begin(); it != end; ++it)
+    // {
+    //     if (it->first == RecordComponent::SCALAR)
+    //     {
+    //         this->container().erase(it);
+    //         throw error::WrongAPIUsage(detail::NO_SCALAR_INSERT);
+    //     }
+    // }


include/openPMD/backend/BaseRecord.hpp

+    // for (auto it = this->container().begin(); it != end; ++it)
+    // {
+    //     if (it->first == RecordComponent::SCALAR)
+    //     {
+    //         this->container().erase(it);
+    //         throw error::WrongAPIUsage(detail::NO_SCALAR_INSERT);
+    //     }
+    // }


test/CoreTest.cpp

@@ -1353,3 +1378,44 @@
    UniquePtrWithLambda<int[]> arrptrFilledCustom{
        new int[5]{}, [](int const *p) { delete[] p; }};
 }
+
+TEST_CASE("scalar_and_vector", "[core]")


test/CoreTest.cpp

@@ -156,6 +159,39 @@
    }
 }

+TEST_CASE("custom_hierarchies", "[core]")


test/CoreTest.cpp

@@ -156,6 +159,129 @@
    }
 }

+TEST_CASE("custom_hierarchies", "[core]")


franzpoeschel · 2023-06-19T09:22:25Z

comment removed, updated version in comments below

test/CoreTest.cpp

src/CustomHierarchy.cpp

franzpoeschel · 2023-07-13T12:53:54Z

For the meshesPath (equivalently for particlesPath), I have now implemented a prototype that does the following:

A path /data/0/custom/group/meshes/E is a mesh if the meshesPath contains any of the following:

Full path to the group containing the mesh: /custom/group/meshes/
Full path to the mesh itself: /custom/group/meshes/E
Shorthand notation: meshes/

These options follow the following rules:

Full paths are denoted by a leading slash and are based on the data path (/data/%T)
Paths that don't specify the mesh itself, but instead its parent path (e.g. meshes/) are denoted by a trailing slash

Remarks:

The shorthand notation achieves backwards compatibility with old openPMD files
A shorthand notation for the meshes themselves (e.g. meshesPath = ['E', ...]) is not implemented, that is on purpose. Regexes can be used instead if someone really wants to have that semantic (.*/E).

franzpoeschel · 2023-07-13T13:11:50Z

One nontrivial design question is how to deal with the traditional openPMD hierarchy, especially with the paths /data/%T/meshes and /data/%T/particles. There is no definition of any form of physical data for those groups in the openPMD standard, a normal openPMD file contains no attributes /data/%T/meshes/<attr_name>.

This suggests to me that in the extended openPMD standard with custom hierarchies these paths should be treated as "nothing special". Rather, they become the canonical, but not mandatory layout/organization of a simple openPMD dataset.

Two somewhat tricky consequences from this point of view:

1. There might be more than 1 meshes paths in the same group
E.g. the paths /data/%T/meshes and /data/%T/images might exist side by side. In the openPMD standard, this is no problem, in the openPMD-api this becomes challenging.
The problem is with the member Iteration::meshes (made even worse by the fact that it's not a getter method, but a data member). Should it point to /data/%T/meshes? To a union of both? What about writing?

Imo, the best solution is to consider Iteration::meshes a shorthand API that should not be used in more complex setups. Rather, since /data/%T/meshes is now just another normal path in the custom Iteration hierarchy, one should access iteration["meshes"].asContainerOf<Mesh>() for clarity.

Iteration::meshes will point to the first user-specified meshes path that takes the form of a shorthand notation. E.g., after series.setMeshesPath({"fields/"}), the call iteration.meshes will be the same as iteration["fields"].asContainerOf<Mesh>(). This ensures backwards compatibility.

(Note: Since Iteration::meshes is unfortunately a member and not a method, this means that the meshes path must be set before creating or opening any Iteration. And it was enough fighting with pointers to get things to that state.)

2. There might be custom data inside /data/%T/meshes
This is not really a problem, but could be unexpected. When setting series.setMeshesPath({"/meshes/E"}), you state that only the E field is a mesh. Since /data/%T/meshes is otherwise "just a regular group" with no special meaning, there might be other data in there, too, e.g. /data/%T/meshes/custom/hierarchy. It's the job of the user to create a meaningful data layout here.

src/CustomHierarchy.cpp

test/CoreTest.cpp

+    }
+}
+
+TEST_CASE("custom_hierarchies_no_rw", "[core]")


Introduction of iteration["meshes"].asContainerOf<Mesh>() as a more explicit variant for iteration.meshes.

Overload resolution

TODO: Since meshes/particles can no longer be directly addressed with this, maybe adapt the class hierarchy to disallow mixed groups that contain meshes, particles, groups and datasets at the same time. Only maybe though..

The have their own meaning now and are no longer just carefully maintained for backwards compatibility. Instead, they are supposed to serve as a shortcut to all openPMD data found further down the hierarchy.

for more information, see https://pre-commit.ci

franzpoeschel added discussion api: new additions to the API feature request labels May 3, 2023

github-advanced-security bot found potential problems May 3, 2023

View reviewed changes

franzpoeschel force-pushed the topic-custom-hierarchies branch from 6c7f23a to c692dc7 Compare May 8, 2023 09:22

github-advanced-security bot found potential problems May 8, 2023

View reviewed changes

test/CoreTest.cpp

@@ -156,6 +159,39 @@

}

}

TEST_CASE("custom_hierarchies", "[core]")

Check notice

Code scanning / CodeQL

Unused static function Note

Static function C_A_T_C_H_T_E_S_T_4 is unreachable (

autoRegistrar5

must be removed at the same time)

franzpoeschel force-pushed the topic-custom-hierarchies branch 2 times, most recently from c8a68a5 to 6c87958 Compare May 11, 2023 09:19

franzpoeschel force-pushed the topic-custom-hierarchies branch 2 times, most recently from 86d8a73 to 399e6cd Compare May 30, 2023 12:43

github-advanced-security bot found potential problems May 30, 2023

View reviewed changes

test/CoreTest.cpp

@@ -156,6 +159,129 @@

}

}

TEST_CASE("custom_hierarchies", "[core]")

Check warning

Code scanning / CodeQL

Poorly documented large function Warning

Poorly documented function: fewer than 2% comments for a function of 194 lines.

franzpoeschel force-pushed the topic-custom-hierarchies branch 2 times, most recently from 8c28fab to 605bd55 Compare June 29, 2023 11:11

github-advanced-security bot found potential problems Jun 29, 2023

View reviewed changes

test/CoreTest.cpp Fixed Show fixed Hide fixed

franzpoeschel commented Jun 29, 2023

View reviewed changes

src/CustomHierarchy.cpp Outdated Show resolved Hide resolved

franzpoeschel force-pushed the topic-custom-hierarchies branch from bef9c6b to b4779a3 Compare July 13, 2023 12:24

github-advanced-security bot found potential problems Jul 13, 2023

View reviewed changes

src/CustomHierarchy.cpp Fixed Show fixed Hide fixed

franzpoeschel force-pushed the topic-custom-hierarchies branch from b0d370e to 4873e21 Compare July 24, 2023 14:34

franzpoeschel force-pushed the topic-custom-hierarchies branch 2 times, most recently from 53f968c to ba10099 Compare August 1, 2023 13:37

github-advanced-security bot found potential problems Aug 1, 2023

View reviewed changes

test/CoreTest.cpp

}

}

TEST_CASE("custom_hierarchies_no_rw", "[core]")

Check notice

Code scanning / CodeQL

Unused static function Note

Static function C_A_T_C_H_T_E_S_T_6 is unreachable (

autoRegistrar7

must be removed at the same time)

franzpoeschel force-pushed the topic-custom-hierarchies branch from ba10099 to d86fa69 Compare August 1, 2023 14:43

franzpoeschel mentioned this pull request Aug 1, 2023

Update Pybind to 2.11.1 #1489

Merged

1 task

franzpoeschel force-pushed the topic-custom-hierarchies branch 4 times, most recently from 31c7a25 to 1d47d17 Compare August 3, 2023 09:25

franzpoeschel force-pushed the topic-custom-hierarchies branch from 795119c to eb2d3c5 Compare February 29, 2024 12:52

franzpoeschel mentioned this pull request Mar 15, 2024

Writing and Reading Non-openPMD Infos #15

Open

franzpoeschel force-pushed the topic-custom-hierarchies branch from eb2d3c5 to 0820e15 Compare March 26, 2024 13:30

ax3l force-pushed the dev branch from b80dfee to 805c760 Compare May 7, 2024 17:28

franzpoeschel force-pushed the topic-custom-hierarchies branch from 0820e15 to 18e9113 Compare May 21, 2024 15:00

franzpoeschel added 23 commits May 24, 2024 15:08

Introduce SharedAttributableData

7306424

JSON backend: Fail when trying to open non-existing groups

b57c36f

Insert CustomHierarchy class to Iteration

22e4a45

Help older compilers deal with this

fe8fbd0

Add vector variants of meshes/particlesPath

4e95eac

Move meshes and particles over to CustomHierarchies class

5ac1788

Move dirtyRecursive to CustomHierarchy

037f563

Move Iteration reading logic to CustomHierarchy

4d63a1c

Move Iteration flushing logic to CustomHierarchy class

6cf1eaa

Support for custom datasets

75ecdce

Treat "meshes"/"particles" as normal subgroups

b7d2cba

Introduction of iteration["meshes"].asContainerOf<Mesh>() as a more explicit variant for iteration.meshes.

Regex-based list of meshes/particlesPaths

6955b54

More extended testing

f2fa126

Fix Python bindings without adding new functionality yet

fcaeaa3

Overload resolution

Add simple Python bindings and an example

b5b8765

Replace Regexes with Globbing

4a07a27

TODO: Since meshes/particles can no longer be directly addressed with this, maybe adapt the class hierarchy to disallow mixed groups that contain meshes, particles, groups and datasets at the same time. Only maybe though..

Move .meshes and .particles back to Iteration class

5b0280b

The have their own meaning now and are no longer just carefully maintained for backwards compatibility. Instead, they are supposed to serve as a shortcut to all openPMD data found further down the hierarchy.

Some fixes in read error handling

4d3a970

More symmetric design for container types

325b347

Don't write unitSI in custom datasets

fd8ecac

Discouraged support for custom datasets inside the particlesPath

95babf3

Fix after rebase: dirtyRecursive

22062d9

Fixes to the dirty/dirtyRecursive logic

fd7a443

franzpoeschel force-pushed the topic-custom-hierarchies branch from 443147c to fd7a443 Compare May 24, 2024 14:37

[pre-commit.ci] auto fixes from pre-commit.com hooks

033ca9f

for more information, see https://pre-commit.ci

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Custom Hierarchies #1432

[WIP] Custom Hierarchies #1432

franzpoeschel commented May 3, 2023 •

edited

franzpoeschel commented Jun 19, 2023 •

edited

franzpoeschel commented Jul 13, 2023 •

edited

franzpoeschel commented Jul 13, 2023 •

edited

[WIP] Custom Hierarchies #1432

Are you sure you want to change the base?

[WIP] Custom Hierarchies #1432

Conversation

franzpoeschel commented May 3, 2023 • edited

franzpoeschel commented Jun 19, 2023 • edited

franzpoeschel commented Jul 13, 2023 • edited

franzpoeschel commented Jul 13, 2023 • edited

franzpoeschel commented May 3, 2023 •

edited

franzpoeschel commented Jun 19, 2023 •

edited

franzpoeschel commented Jul 13, 2023 •

edited

franzpoeschel commented Jul 13, 2023 •

edited