Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDF5: Remove Support for Old Versions #1514

Open
ax3l opened this issue Aug 18, 2023 · 1 comment
Open

HDF5: Remove Support for Old Versions #1514

ax3l opened this issue Aug 18, 2023 · 1 comment
Assignees
Milestone

Comments

@ax3l
Copy link
Member

ax3l commented Aug 18, 2023

If we bump up our support to newer HDF5 versions, then we can optimize meta-data features & performance:

The call H5Pset_libver_bounds will set the guaranteed compatibility with older readers. The less old versions are supported, the more performant we can be.

Cool features in Quincy Koziol's ATPESC23 or HUG23 slides (to be linked):

  • size of metadata structures H5Pset_istore_k, H5Pset_sym_k, etc.
  • metadata aggregation in larger blocks H5Pset_meta_block_size
  • Align objects in the file H5Pset_alignment
  • Metadata cache size
  • Page allocation and page buffering (see: File Space management)

We got to check, but can probably dump everything before 1.12 because our users & readers use recent pip/conda packages that mostly ship HDF5 1.14 already.

cc @jeanbez already increased quite a few of these buffers, but I was unaware of H5Pset_libver_bounds performance implications so far.

@ax3l ax3l added this to the 0.16.0 milestone Aug 18, 2023
@ax3l ax3l self-assigned this Aug 18, 2023
@jeanbez
Copy link
Member

jeanbez commented Aug 23, 2023

@ax3l let me know the minimum version you plan to support so I can open a PR with this. I can check the other optimizations since most have already been applied. I'm curious about their possible impact on openPMD.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants