Skip to content

HDF5 Working Group

Jump to bottom
Dana Robinson edited this page May 30, 2024 · 76 revisions

The HDF5 Working Group meets weekly, on Thursdays at 10 am Central time. This meeting is for HDF5 library developers and anyone is welcome to attend. The purpose of this meeting is to discuss HDF5 library development. It is NOT intended for providing technical support.

Zoom link: https://us06web.zoom.us/j/89601195963

The agenda and any cancellations will be posted below and on the forum, usually on Monday. If you have any action items to discuss, please email derobins at hdfgroup dot org to get them added to the agenda. If there are no pressing issues by Monday, the meeting may be cancelled.

30 May 2024

Agenda

  • CMake discussion
  • Poisoning discussion
  • Should we have an HDF5 1.16.0 release in the fall?
  • What do we do about the C++ wrappers?

CMake Plan

See: https://forum.hdfgroup.org/t/community-input-hdf5-cmake-overhaul/12364/2

For HDF5 1.14.5, we’re planning on overhauling our CMake build code, to bring it more in line with modern CMake conventions.

We’re still putting together our assessment/plan, but we’d like to:

  • Consolidate and reorganize the existing CMake code, to make things easier to understand and find
  • Move to “modern CMake”
    • Convert macros to functions
    • Add additional functions to avoid code duplication
  • Ensure parity with the Autotools
  • Add verification/testing to CI to ensure builds are correct
  • Add documentation

Useful "modern CMake" links

Poisoning discussion

  • Adding a way to "poison" the library so subsequent API calls will just fail and not mutate state
    • Helpful for debugging
    • Could avoid making problems worse when library state gets corrupted
    • Concurrency is the driving force for this
  • Would be mid-way in impact between assert/abort and just returning an error code
  • Could be implemented easily via a global "poisoned" variable that is checked on API entry
  • Consensus is "maybe, but let's think about it"
  • Should probably be off by default, at least in release builds
  • We should set up an RFC/PIP process for HDF5 so things like this can be documented and discussed

HDF5 1.16.0 in fall 2024

  • Complex number support requires bumping a datatype version number and that should happen in a major release
  • Should we do a major release (probably 1.16.0) in the fall to get complex number support out?
  • Tentative 'yes' and we'll start planning (more news at the June 3 Call the Doctor)
  • Might bump some library defaults (e.g., to better work in the cloud)
  • Could make minor API changes (e.g., #3505 HDoff_t change)
  • Should think about bumping the minimum CMake version to get workflows (3.25?)
  • Might also think about making some build system changes (drop obsolete things, rename for consistency)
  • Probably not move to C11 to make the upgrade path easier, but maybe bump develop to C11 after creating the 1.16 branch
  • Would try to make upgrading from 1.14.x as easy as possible
  • Would no longer support 1.14.x (so 1.14.4 would be the last 1.14 release)

C++ Wrappers?

  • These have lagged far behind the library, feature-wise
  • Do we update them or deprecate and eventually remove them? There are header-only C++ wrappers that provide a more modern interface (not maintained by HDFG)
  • We'll ask the community (forum posts, CtD, state-of-HDF5 talks)

23 May 2024

Agenda

  • Clean out PR queue

Actions

NOTE: In general, PRs that will never be merged but contain useful ideas should be closed and an issue/discussion created instead. PRs that are a work in progress, are under active development, and should not be merged until important changes have been made should be marked draft while the work takes place. Everything else in the PR list should be ready for review and possible merging.

  • 1387 - (H5T optimization) - This PR will never be merged and was being kept as a reminder to investigate the useful bits of the PR. We will close it and create an issue.
  • 3505 - (HDoff_t) - This PR or an equivalent will be merged before June. It cannot go to 1.14 since it's an API change (which will be unversioned). Needs better docs and probably an issue to ensure docs are correct in next major release.
  • 4171 - (NVHPC update) - Will remain open while we investigate why long double conversions are failing.
  • 4266 - (uninit H5T memory) - Close and create an issue. Needs to be fundamentally fixed elsewhere.
  • 4315 - (datatype precision overflow) - Leave open. Needs minor tweaks.
  • 4347 - (vasprintf) - Leave open. Will be converted to use HD prefix.
  • 4469 - (big threading PR) - Needs review. Copyright okay now.
  • 4475 - (pause error checking) - Okay to review and merge. Needs an issue to update the docs, especially the VOL connector author guide. Also probably needs VOL connector upgrades. Might be a 2.0 thing.
  • 4487 - (zlib-ng) - Can remain open while we investigate the test failure.
  • 4488 - (URLs) - Can remain open as a draft but needs a plan for resolution.
  • 4500 - (CMake UNITY_BUILD) - Review and merge.

16 May 2024 - CANCELLED (no agenda items)

9 May 2024

Agenda

  • Make sure everyone can connect to Zoom
  • Complex number datatype creation API
    • Does it make sense to allow specifying member names for real and imaginary part of complex numbers and store that in the file?
    • Should we design the API to allow different datatypes for real and imaginary part, based on forum discussion?
  • When/if C11 is moved to, alignment checking of C types in library can be replaced with _Alignof(type) / alignof(type) (latter removed in C23)
  • Additional locking discussion (Quincey Koziol)
  • PRs and such

Complex number datatype discussion

  • Does it make sense to allow specifying member names for real and imaginary part of complex numbers and store that in the file?
    • Use case is mainly display
    • Could be contentious
    • Might be better to push this to the tools
  • Should we design the API to allow different datatypes for real and imaginary part, based on forum discussion?
    • Is there a compelling use case for this?
  • Jordan will push the feature to a branch soon
  • Could make complex numbers a datatype class and select between cartesian/polar
  • Alternatively, just support C complex type (like how the rest of the library basically works)

When/if C11 is moved to, alignment checking of C types in library can be replaced with _Alignof(type) / alignof(type) (latter removed in C23)

  • It makes sense to allow this when we move to C11 (in next major version?)

Climate workshop (Elena)

  • Switched to Zarr from HDF5 due to lack of SWMR
  • LZ4 filter function issues - chunk can't be uncompressed directly

Zarr discussion

  • Need to address MT, etc. concerns
  • We should support FP8, which Zarr does not

Additional locking discussion (Quincey Koziol)

  • Sent along a doc (in chat)

PRs and such

2 May 2024

Agenda

Move to Zoom (starting next week)

  • Moving to Zoom so it's easier for outside people to join
  • Agenda will be listed here and announced on the forum on Mondays
  • I'll send a Zoom link to everyone who is on the existing Teams invite list
  • Might use one of the meetings per month for the filter working group

Governance

  • Will be looking at other projects to get some ideas for how to better manage the HDF5 community

PRs and such

  • Do we check in generated files? (#4453, which commits newly-added, generated files) - Quincey suggests having a script to generate the files as a convenience
  • Should the codestack removal go to 1.14? (#4454) - No strong feelings, definitely broken and nobody complained, just a configure change, let's remove

Removing Autotools support

  • Supporting two build systems is a lot of work
  • libtool is a mess and requires sed hacks to fix linker options (#4448)
  • How much screaming will there be if we drop the Autotools in the next major version of the library?
  • What about Ubuntu, etc.?
  • Look for complaints in the forum

Is it okay to allocate file space when there is a null dataspace

  • An oversight, nothing gets allocated at zero bytes
  • Need to update the docs

Discuss locking protocol for multithreaded concurrency planning

25 April 2024

Agenda

  • PRs and such
  • Property lists (Lifeboat)

PRs and such

  • FUNC_ENTER has some controversies and needs investigation
  • Others can go in after review requirements are met

Property lists (Lifeboat)

  • Do we need to support modifying a property list class after properties exist? (yes)
  • Do we need to support deleting properties from a class? (maybe)
  • Existing library is buggy in that properties can be deleted from a property list class and this will affect existing property lists (John Mainzer will file an issue)

18 April 2024

Agenda

  • Yay, release
  • C11 in develop
  • Issues, etc.
  • Revisions to the error package (H5E)

Yay, release

  • It's out. Yay!

C11 in develop

  • Is it okay to require C11 in develop?
  • Tentative yes, but we need a policy
  • Also need to figure out where we are going to test and debug big-endian code

Issues, etc.

  • Not a lot going on
  • Let's get the H5Tconv refactoring in

Revisions to the error package (H5E)

  • NVidia thread-safety changes passing CI, PRs imminent
  • Trying to reduce H5I usage from H5E
  • H5E uses H5I for its data management, should revamp to store things in local data structures
  • Legit use cases for creating custom ID things (e.g., VOL connectors)
  • Will still use IDs, but only for public new classes that will be immediately converted to the internal things
  • So the public API will not change
  • Library internals should be refactored to avoid ID use internally, in general
  • Lifeboat says this is fine wrt what they are doing

11 April 2024

Agenda

  • Release goings on
  • (Pseudo-)random numbers in HDF5
  • What do we do with the last HDfoo() functions?
  • Do we keep the 'getting started' guide and similar docs in the GitHub wiki or with the code?
  • H5TRACE scheme removal
  • H5E changes && "have threads" vs "have threadsafe" macros (Quincey Koziol)

Release goings on

  • Release is TODAY 🎉

Pseudo-random numbers

What do we do with the last few HDfoo() functions?

Where to put helpful developer docs?

H5TRACE scheme removal

H5E changes

  • Quincey wants to not incr/decr library error IDs
  • Maybe cache error IDs in a table?
  • Will require further investigation

"have threads" vs "have threadsafe" macros

  • H5_HAVE_THREADS --> Has threads
  • H5_HAVE_THREADSAFE --> Has thread-safety (i.e., global lock)

4 April 2024

Agenda

  • Release goings on
  • Locking protocols discussion (Quincey Koziol)

Release goings on

Locking protocols discussion

28 March 2024

Agenda

  • Release status
  • #3505: off_t --> HDoff_t
  • Quincey's threadsafety wrapper work
  • Do we mandate C11? (at least in develop)
  • PR/issue highlights
  • Crashproofing RFC

Release status

  • Delayed until early/mid-April (waiting on a security patch from Amazon)

#3505: off_t --> HDoff_t

  • We did not version anything when we went from hid_t int --> int64_t in 1.8/10
  • Let's merge this PR (can't go to 1.14)
  • This is fine
  • Quincey sees off_t issues on his Mac w/ gcc 13

Do we mandate C11? (at least in develop)

PR/issue highlights

  • Big set of changes from Quincey / AWS has been merged (thanks!)
  • NOTE: These will be filed as Mitre CVEs

Quincey's threadsafety wrapper work

Crashproofing RFC

  • John Mainzer insists that VFD SWMR has many of these features
  • Will send out the RFC to meeting attendees
  • Will schedule a follow-up meeting to discuss
  • Neil will present at next week's call the doctor meeting

21 March 2024

Agenda

  • Upcoming release
  • PR/issue highlights

Upcoming release

  • Try to get your code in by Friday (March 22)
  • Release will be next Thursday (March 28)

PR/issue highlights

  • Issue #108 - Needs a file format bump, might want to boost to 64-bit chunks, could do as a part of sparse work
  • PR #4166 - Jordan will close, we'll revisit for 1.14.5 (possibly a doc change)

14 March 2024

Agenda

  • float16 goings on
  • PR/issue highlights

float16 goings on

  • Will merge PR on Friday

PRs and Issues

  • Not much exciting going on
  • John may have H5I lock-free code for us to look at in April
  • Mark can help us with HDF Java --> Maven in April
  • Quincey talked about the need for H5I locks even with lock-free data structures

7 March 2024

Agenda

  • float16 goings on
  • The TRACE macros and related
  • PR/issue highlights

float16 goings on

  • Going pretty well
  • Dealing with quirky platforms/compilers
  • Scot says there are some Fortran issues with Flang (REAL16 support is in Fortran 2023 - Flang is the only compiler that supports this (currenty))

TRACE macros

  • Discussion
  • Thoughts?
  • Jordan: Never found it to be useful at all
  • Decision is to merge existing PR, then tag in GH, and then create a PR to yank

PR/issue highlights

  • (#4070 - BE examples fail) What do we do about examples?
  • Jordan says h5ls and h5dump have opposite ideas about printing types (we should fix this)
  • (#4064 - ros3 VFD secret length) Can go in. Dana will fix the stack allocations later.
  • (#4062 - static CRT link) Can go in. Dana will update the PR to have a "don't use this!" comment
  • (#4060 - long double test) Can go in

29 February 2024 (No meeting - USENIX FAST)

22 February 2024

Agenda

  • PR #4017 (improves flush performance) <-- Click merge button if no objections
  • float16 goings on
  • Other library happenings

15 February 2024

Agenda

  • PR #4017 (improves flush performance)
  • Other library goings on

Notes

Flush performance PR

  • Looks good at a first glance
  • We'll discuss next week and merge after the HDF5WG

GitHub goings on

  • FAPL changes will propagate to sub-files in things like VDS, external links
  • Should change the behavior of H5Pset_elink_fapl() so you get the propagating behavior and only have to use the API call if you want something different

Quincey's NVidia todo list

  • ~20 project ideas regarding GDS, etc.

8 February 2024

Agenda

  • Float16 RFC update
  • Chat about chunk cache meeting

Notes

Float16, etc.

  • Brief discussion of complex datatypes, will keep on working on the RFC

Chunk cache discussion

  • Will kick this down the road until there are phase II funds (April?)

GitHub goings on

  • Need to have ABI checks when merging to 1.14
  • Need to be able to easily find ABI check results for develop, 1.14
  • Are build-only checks running tests?

Misc questions

  • Do the Autotools + gcc strip -g (or not pass it) to the compiler, linker, etc. ? Is -s getting passed down and superseding -g? John M. will hassle Dana if --enable-symbols doesn't work.

1 February 2024

Agenda

  • Float16 RFC update
  • Citing HDF5 PR
  • HDF5 goings on
  • Debug API?

Notes

New types: float16, complex, Boolean

  • Do we need a recap of type conversion in the RFC?
  • We should definitely have examples in the docs for when a particular type is not supported
  • What about quad precision floats (128-bit, supported in GNU, numpy and Fortran already support, there's an IEEE standard)?

Citing HDF5 PR

  • Merged

HDF5 goings on

  • Nothing too exciting

Debug API

  • General agreement that some low-level API calls could go in a special debug header that would not be a formal part of the 'public API'

25 January 2024

Agenda

Notes

GCPL propagation

  • We'll propagate the properties to the intermediate groups
  • Might require some finagling to deal with non-group creation

New types: float16, complex, Boolean

  • We should look at FP8 as well, possibly other ML/DL convenience reduced types
  • Should also consider adding Boolean support from Jerome's RFC
  • Need to look at NVidia's compilers
  • NATIVE macros will map to H5I_INAVLID_HID when there is no compiler support for a type
  • Complex should be a real atomic type, not a struct of two doubles or whatnot
  • Might add some conversions to common current complex number hacks, like a struct of two doubles, etc.
  • How will complex numbers be printed in h5dump, etc.?

18 January 2024

Agenda

  • Discuss how the chunk cache discussion went
  • Library PRs, issues, etc.

Notes

Discuss how the chunk cache discussion went

  • John Mainzer / Lifeboat will be reworking the chunk cache for sparse data and thread-safety
  • The chunk cache used to be per-file (currently per-dataset - performance was poor w/ per-file)
  • The per-file --> per-dataset change was in 1.6
  • John wants to go back to one cache per file or even one cache for many files
  • John is exploring data structures and algorithms (lock-free hash tables suitable?)
  • Actual work would only take place if Lifeboat gets a phase II grant (end of Feb)

Library PRs, issues, etc.

  • Jordan's type conversion fix is super important and needs to get into 1.14.4. Quincey will review.
  • The -shared CMake extensions for tools, libraries, etc. will be removed by default in 1.14.4. CMake will work more like the Autotools, though compatibility options will be provided if you want the old behavior.

11 January 2024

Agenda

  • Discuss https://github.com/HDFGroup/hdf5/pull/3927
  • Determine 2024 / 1.14.4 priorities
  • Quincey's warnhist improvements
  • John Mainzer has a question about chunk cache documentation
  • Elena Pourmal wants to discuss behavior when a compressed buffer is bigger after "compression"
  • We need an updated guide for updating across major/minor releases
  • Always need to reclaim vlen memory, even when using strings?

Notes

Citing HDF5

  • Tentative okay on #3927, but we should look into how other software does things. Is this the way people will do this in the future?
  • Do we have a DOI for HDF5? That should be a thing we add.

Priorities

  • Unicode
  • MinGW (especially w/ MSYS2 - AND DOCUMENT AND TEST)
  • Update to handle recent MPI session changes
  • Support modern C and ML/DL data types (float16, bool, complex, etc.)

Behavior when a buffer is bigger after "compression"

  • Existing behavior: Larger buffer is written out as "compressed"
  • Responsibility lies with filter
  • Need to fix the zlib deflate filter (Elena will file a bug report, may also affect szip)

Changes to bin/warnhist

  • Now can produce warning density and has more options

Chunk cache documentation?

  • Read the source, Luke
  • Rob Matzke may have an old document (location?)
  • John wants to discuss creating a more flexible chunk cache that can support sparse data better (will be in Champaign T/W next week)

Version updating

  • Version on web has dead links, is out of date

Vlen string cleanup

  • h5py not doing a great job at cleaning up after itself, has a memory leak
  • h5py needs updating to avoid the leak
  • Is h5py doing the right thing with new-style references? <-- New-style references in general probably need better documentation
Clone this wiki locally