Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TDM schema hardening #268

Open
brettviren opened this issue Dec 19, 2023 · 0 comments
Open

TDM schema hardening #268

brettviren opened this issue Dec 19, 2023 · 0 comments

Comments

@brettviren
Copy link
Member

This is a low-priority wishlist todo reminder for when someone is bored.

The tensor data model schema and I/O methods should be "hardened". Right now we hand-craft I/O converters between higher level WCT structures (ICluster, PointCloud::Dataset, etc) and file representations. It requires careful human attention at every point to develop and maintain.

Better would be to add a subsystem that goes something like this:

  1. Formally represent the TDM schema with moo schema.
  2. Develop a C++ moo template file operating on a TDM moo schema object that renders to a C++ struct representing that TDM object.
  3. Develop a C++ moo template file operating on a TDM moo schema object that generates I/O methods converting between the above struct and a file representation. One template targeting the WCT json+numpy+archive format and one targeting hdf5.
  4. Develop a C++ moo template operating on a collection of all TDM moo schema objects that generates the necessary dynamic-static type conversion needed to serialize a file of multiple TDM records of differing types.

See #267 for recent work in this area and in particular not the use of the located index which is required to speed up I/O operations. This index might live as part of the generated code in 3 or 4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant