Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

I'd like a function to produce a frame epilogue on the final frame of a stream. #2090

Closed
xorgy opened this issue Apr 24, 2020 · 3 comments
Closed
Assignees

Comments

@xorgy
Copy link

xorgy commented Apr 24, 2020

Is your feature request related to a problem? Please describe.
I'm appending synchronously to a file, flushing every time I write something, but I don't want to write a frame epilogue on every write because it will kill my compression advantage. I've noticed that there is no particular problem decoding a properly-flushed stream with no epilogue on the final frame, except that the decoder warns, and you can no longer safely append to it.

Describe the solution you'd like
I would like some straightforward way to append a valid frame epilogue on a stream that was flushed, but not closed properly, so that I can continue to append to the stream rather than having to recompress the whole thing (or in my application's case, start a new chunk).

Describe alternatives you've considered
I've considered simply recompressing when encountering a stream in this state, since it would be relatively rare, but if I encounter it more often, I will want a way to accomplish this without recompressing.

@terrelln
Copy link
Contributor

@xorgy a function like this should work (definitely test it), but only without the frame checksum enabled, since it would be impossible to compute without decompressing the entire stream. See the frame format https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#blocks. I'm not sure that we really want to add this to our API, but the frame format is fixed, so you could implement it locally without risking incompatibility.

// Writes the last block, returns -1 if the output buffer is too small
int ZSTD_emitEmptyLastBlock(void* dst, size_t size) {
    // Last block bit set, uncompressed block, size 0.
    char const kLastEmptyBlock[3] = {0x01, 0x00, 0x00};
    if (size < sizeof(kLastEmptyBlock)) return -1;
    memcpy(dst, kLastEmptyBlock, sizeof(kLastEmptyBlock));
    return 3;
}

@terrelln terrelln self-assigned this Apr 27, 2020
@xorgy
Copy link
Author

xorgy commented Apr 27, 2020

I'll give something like that a shot, thank you. Barring that, I could probably write what I wanted as an external tool.

@terrelln
Copy link
Contributor

Please reopen if you have further questions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants