Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

how to use checksum? #237

Open
erenturkm opened this issue Aug 19, 2023 · 1 comment
Open

how to use checksum? #237

erenturkm opened this issue Aug 19, 2023 · 1 comment

Comments

@erenturkm
Copy link

Hi,

I am using zstd for very large files and checksum feature is important for my use case. I need more information on how to use this feature:

  1. When checksum is included from the encoder, how is it calculated?
  2. For stream of a single file, does this mean there is a single checksum or is there multiple checksums for different parts? where is it stored?
  3. How do I check if a file is intact?
  4. does decoder automatically check checksum if there is one?
@gyscos
Copy link
Owner

gyscos commented Oct 11, 2023

  • The zstd_safe::CParameter::ChecksumFlag can be used when compressing to include a single 32-bit checksum (of the original, uncompressed content) per frame (at the end of the frame). In general there is one frame per file, but that's not enforced. It's possible for some files to be a series of zstd frames (by default the decoder just concatenates the decompressed content); each frame would have its own checksum.
    The checksum uses xxh64 (from xxHash).
  • When decompressing, checksums are automatically checked if present.
  • To "test" a file, you can "decompress" it and throw away the result; it will check that the decompressed content matches the checksum and return an error if it doesn't.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants