Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide decompressed file size #233

Open
djugei opened this issue Jul 27, 2023 · 2 comments
Open

Provide decompressed file size #233

djugei opened this issue Jul 27, 2023 · 2 comments

Comments

@djugei
Copy link

djugei commented Jul 27, 2023

to my best understanding zstd has an optional header field telling us how big the uncompressed size is,
alternatively you could sum up all frames?

it would be nice to have
Decoder::size_hint(&self) -> Option <u64>

@gyscos
Copy link
Owner

gyscos commented Oct 11, 2023

Hi, and thanks for the report!

With the experimental flag enabled, there are two functions you can use:

  • zstd_safe::get_frame_content_size: given a prefix of the frame (at least ~5 bytes long), returns the decompressed size of this frame.
  • zstd_safe::find_decompressed_size: given the entire compressed content (could be a series of several frames), returns the total decompressed size (the sum of the decompressed size of each frame).

These are not available on Decoder itself - a Decoder could be in the middle of streaming, and have no memory of the header, and no knowledge of future frames in the stream.

We could at least re-export these methods in zstd itself.

@djugei
Copy link
Author

djugei commented May 2, 2024

progress: zstd::bulk::Decompressor::upper_bound does mostly what i need. issues with it:

  • interface is wrong, returns an option, but its only none if a compile time feature is missing, imo it should return a straight value
  • heavily overestimates on files created without a size header (for example by piping), probably unfixable
  • needs access to a &[u8], basically forcing the use of mmap, this does not fit zstds usual file/writer/reader-based api

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants