Read data while concurrently writing it (SWMR - SingleWriter / MulipleReader)

[eachristgr]

Hi!

I want to be able to simultaneously write and read an HDF5 file.

HDF5 1.10.0 and newer versions support the Single-Writer / Multiple-Reader or SWMR feature enables users to read data while concurrently writing it (Announcing HDF5 1.10.0).

I know that python library h5py supports this feature (h5py SWMR), but does HighFive support this? Do you plan to implement this feature in the near future? If not, how difficult do you think it is to develop?

[1uc]

From briefly reading the HDF5 documentation it seems that the process requires:

• Opening/creating an HDF5 file. One might need to state that one wants the modern file format. Hi5 can be used for all of this.
• Create the required structure in terms of groups and datasets. The dataset you want to grow must be unlimited along the direction you want it to grow. To the best of my knowledge you can do all this in Hi5.
• You'll need to mark the file as SWMR: https://support.hdfgroup.org/HDF5/doc/RM/RM_H5F.html#File-StartSwmrWrite. The signature is:

  herr_t H5Fstart_swmr_write(hid_t file_id);
  

  You can get the file_id from a Hi5 file.
• For writing the first thing we find is

  herr_t H5DOappend( hid_t dset_id, hid_t dxpl_id, unsigned index, size_t num_elem, hid_t memtype, const void *buffer );
  

  This is mixed news, good thing is you don't need to specify the shape, hence everything simplifies down to a number of elements, type of elements and the address of the first byte. You can get the dset_id from the Hi5 dataset and you can maybe create the dxpl_id through Hi5. However, reading the documentation suggests that H5DOappend is short-hand for extending the dataset, selecting the part you want to write to and actually writing. As far as I know all three steps are supported in Hi5.
• The readers will need to poll the file using

  herr_t H5Drefresh(hid_t dataset_id);
  

  again you can get the dataset_id from the Hi5 dataset. See,
  https://support.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-Refresh
• Afterwards readers can get the updated size of the dataset and read as usual. Meaning it should be possible to read the updated dataset using Hi5 as normal.

To me it looks like the heavy lifting can be done using Hi5 and you'll only need to call two C functions which have a seemingly harmless API. Supporting SWMR would certainly be nice and it seems like not much is missing: two trivial wrappers and means to append to a dataset along a given axis in one line of code. Currently, I don't see this being added to Hi5 in say the next week or two.