Write to HDF5 (optionally read) but from C++

[HealthyPear]

Description of new feature

Hello,

I have found this in your docs
https://awkward-array.org/doc/main/user-guide/how-to-convert-buffers.html?highlight=hdf5#saving-awkward-arrays-to-hdf5

which I knew about, unfortunately I am working from a C++ project and since I saw that it should be possible to use this library directly in C++
https://awkward-array.org/doc/main/user-guide/how-to-use-header-only-layoutbuilder.html

I was wondering if it was possible to have the same tutorial but from C++ instead of Python.

I tried to go through HDF5 docs and a bunch of other libraries of C++ interfaces but I didn't get out much useful info...

[jpivarski]

The HDF5 example in the docs is not a supported format so much as a demonstration. The reason it's not considered a format is because someone could open the HDF5 file and not know what to do with the array buffers; the Awkward library would be needed to make sense of it, and there's no indication in the HDF5 file itself to say that one should use Awkward to do that.

On the other hand, a format like Parquet does describe itself as ragged and nested arrays, rather than flat buffers that need an interpreter. If you're producing data in C++, perhaps you'll have an easier time using the Parquet C++ tools to write to Parquet files, and then read them in Python using Awkward.

The HDF5 example isn't code, it's a demonstration of how to put buffers from the ak.to_buffers function into an HDF5 file as arrays. The LayoutBuilder to_buffers method creates the same kind of buffers that could, in principle, be written to an HDF5 file in C++. There are HDF5 interfaces in C and in C++ that could be used for that, but I'd strongly recommend Parquet.

...Unless your data is strictly rectilinear arrays! In that case, just use HDF5 as-is, without anything from Awkward Array, and read it with h5py (with NumPy → Awkward using ak.from_numpy.

[HealthyPear]

Sorry for the delay!

I am not an expert in data format, so I am probably missing some details that might make my life easier.

My use-case is one in which I am writing a piece of software based on Geant4 which at some point should write data as a table with ragged arrays. In a previous-generation code the data format from which data was written in C++ was XCDF, but I wanted to use a more flexible data format that would allow me to save complex metadata together with the list of output events row data (e.g. a table or a histogram of simulated counts) and I know HDF5 allows to do that because it is essentially a small isolated filesystem (and Geant4 has API to write to HDF5.

Does Parquet allow this (from [here](https://parquet.apache.org/docs/file-format/metadata/ it seems to me it should be very similar)? Then reading this kind of data in Python (specifically the list of XCDF-like events) would result in a single "awkward" table (see #2597 and #2468 I'm the same guy 😅 ).

thanks!