Compression and chunk size outside of H5Easy

[ischoegl]

First of all, thanks for making this available - I just finished porting some code that used Python's h5py to HighFive and it was pretty straightforward. One thing that I haven't figured out yet, however, is how to set chunk size and compression level from HighFive. I noticed that interfaces exist and are documented for H5Easy, but I haven't found much for HighFive itself.

I am currently writing data with type std::vector<double> to a HighFive::File that's already open as follows:

    h5::Group sub = m_file.getGroup(id);
    std::vector<size_t> dims{data.size()};
    h5::DataSet dataset = sub.createDataSet<double>(name, h5::DataSpace(dims));
    dataset.write(data);

I assume that I need to set some properties on dataset but am a little lost. For chunks, there appears to be a class (HighFive::Chunking), but I only found a reference to compression for H5Easy. Any pointers would be truly appreciated!

Edit: I found an example of Chunking in create_extensible_dataset.cpp, but am still unsure about compression

PS: For compression, it looks like I may be able to just use the bits I found in H5Easy_misc.hpp

[1uc]

In core HighFive it's called Deflate or SZip. You'd probably do something like:

auto file = HighFive::File(...);
auto dcpl = HighFive::DataSetCreateProps{};
dcpl.add(HighFive::Chunking(...));
dcpl.add(HighFive::Deflate(n));

// If you like, you can fuse creation and writing of the data set
// into a single operation:
sub.createDataSet<double>(name, data, dcpl);

Upstream documentation suggests that "deflate" refers to "zlib", see
https://docs.hdfgroup.org/hdf5/develop/_f_i_l_t_e_r.html

[kdidkovsky]

Is there any info/documentation on properties that can be added to DataSetCreateProps? Doxygen-generated docs (https://bluebrain.github.io/HighFive/namespace_high_five.html#afc468b8dcc348886b27bc4edba46caf3) only say that it's a property list, but I couldn't find any documentation on the properties itself.

[1uc]

You can scroll down this file:
https://github.com/BlueBrain/HighFive/blob/master/include/highfive/H5PropertyList.hpp

However, we've only wrapped a small subset of all available properties. Hence, you'd likely first look at HDF5 to understand what's available and what you need to accomplish you goals. Then you can check if it's wrapped in HighFive, e.g. by searching Github for the HDF5 function name.