New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Zstd compression #357
Comments
This should be relatively easy to add. |
I should point out that the plugins in HDF5Plugins.jl are implemented in Julia, having been ported from C. They depend on the Codec* packages but mainly use them for C library wrapper. For example, the Zstandard plugin implementation is found here: Also note that we are merging the plugins directly into the HDF5.jl git tree: |
Looking at how the current compression filters are implemented, I'm not sure if they are compatible with the way the HDF5 plugins work. For example, for LZ4 compression the HDF5 plugin stores the original size of the file as well as the blocksize before storing the compressed data. https://github.com/JuliaIO/HDF5Plugins.jl/blob/main/src/H5Zlz4.jl |
Hi @mkitti , it's great to see you taking an interest in this.
This is important. JLD2 compression tries to conform to the HDF5 standards. Thus JLD2 files using Bzip2 and Zlib compression can be decoded by It'd be very neat, if one could transfer the knowledge gained from |
@JonasIsensee bumping this up, I still get a ArgumentError: Unsupported Compressor
Supported Compressors are
JLD2.ShuffleFilter
CodecBzip2.Bzip2Compressor
CodecLz4.LZ4FrameCompressor
CodecZlib.ZlibCompressor when using |
Shouldn't be much work at all. Lines 51 to 66 in ccd60eb
and the corresponding filter ID, you can look up at:
|
Great, I'll create a PR! |
You can use this as a reference: |
The new registered filter plugins reference page is here: https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md |
I noticed it done here: https://github.com/JuliaIO/HDF5Plugins.jl/blob/main/Project.toml
I noticed in this package Blosc is commented out, for some reason (also not in the package above/main HDF5), not sure why, I don't really care if it's supported if the best format is support that I think is Zstd (seems also important for compatibility with main format):
JLD2.jl/src/compression.jl
Line 13 in f7a946b
I also commented on another closed PR (on pointers to null), regarding a typo (and slipped in a question). I hope people see it, not sure if they do since it's closed. Another question, do (Ptr) nulls take up space in the output stream?
The text was updated successfully, but these errors were encountered: