Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is it splittable? #323

Closed
robert3005 opened this issue Sep 1, 2016 · 4 comments
Closed

Is it splittable? #323

robert3005 opened this issue Sep 1, 2016 · 4 comments
Labels

Comments

@robert3005
Copy link

Can I partition the compressed file and read it in chunks or does the compression format encode whole file

@Cyan4973
Copy link
Contributor

Cyan4973 commented Sep 1, 2016

The format allows multiple frames to be appended.
They will be regenerated in the same order, as if it was a single content.

So yes, it makes it possible to cut files into chunks of any size.

@terrelln
Copy link
Contributor

terrelln commented Sep 1, 2016

You can break the original file up into chunks, compress each chunk independently into a Zstandard frame (using the zstd CLI), and concatenate each frame. This will still be valid in the eyes of the Zstandard format. You will then be able to break the compressed file up into the component frame and decompress each frame independently.

The only problem is that Zstandard doesn't encode the size of each frame, only the size of each block. So to find where the next frame starts you will have to do one of two things:

  1. Seek through the frame a block at a time until you find the next frame.
  2. Encode some extra information into the compressed data to indicate the size of each frame. You can use a Skippable Frame, as defined by the Zstandard format to do this.

@Abdulmalik13
Copy link

Cod

@Cyan4973
Copy link
Contributor

Issue merged into #395

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants