You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
I'm appending synchronously to a file, flushing every time I write something, but I don't want to write a frame epilogue on every write because it will kill my compression advantage. I've noticed that there is no particular problem decoding a properly-flushed stream with no epilogue on the final frame, except that the decoder warns, and you can no longer safely append to it.
Describe the solution you'd like
I would like some straightforward way to append a valid frame epilogue on a stream that was flushed, but not closed properly, so that I can continue to append to the stream rather than having to recompress the whole thing (or in my application's case, start a new chunk).
Describe alternatives you've considered
I've considered simply recompressing when encountering a stream in this state, since it would be relatively rare, but if I encounter it more often, I will want a way to accomplish this without recompressing.
The text was updated successfully, but these errors were encountered:
@xorgy a function like this should work (definitely test it), but only without the frame checksum enabled, since it would be impossible to compute without decompressing the entire stream. See the frame format https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#blocks. I'm not sure that we really want to add this to our API, but the frame format is fixed, so you could implement it locally without risking incompatibility.
// Writes the last block, returns -1 if the output buffer is too smallintZSTD_emitEmptyLastBlock(void*dst, size_tsize) {
// Last block bit set, uncompressed block, size 0.charconstkLastEmptyBlock[3] = {0x01, 0x00, 0x00};
if (size<sizeof(kLastEmptyBlock)) return-1;
memcpy(dst, kLastEmptyBlock, sizeof(kLastEmptyBlock));
return3;
}
Is your feature request related to a problem? Please describe.
I'm appending synchronously to a file, flushing every time I write something, but I don't want to write a frame epilogue on every write because it will kill my compression advantage. I've noticed that there is no particular problem decoding a properly-flushed stream with no epilogue on the final frame, except that the decoder warns, and you can no longer safely append to it.
Describe the solution you'd like
I would like some straightforward way to append a valid frame epilogue on a stream that was flushed, but not closed properly, so that I can continue to append to the stream rather than having to recompress the whole thing (or in my application's case, start a new chunk).
Describe alternatives you've considered
I've considered simply recompressing when encountering a stream in this state, since it would be relatively rare, but if I encounter it more often, I will want a way to accomplish this without recompressing.
The text was updated successfully, but these errors were encountered: