You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a scenario where I continuously receive streamed data and I want to compress this data in multiple frames with a pre-given FrameSize. I am using the C-API function ZSTD_CCtx_setPledgedSrcSize() for this purpose. The problem arise when the data stream stops or requests closing the file and starting a new file in a time point when the the current frame size is not yet reached. I can't close the frame in this case. I thought I could do padding with meaningless data till the end of the frame. However, this would not be clean when decompressing the file. If one could use something like skippable blocks, then I could fill the rest of the frame with these and the decompressor would skip them automatically. This could solve my problem. Otherwise I would appreciate any help to overcome this issue.
Thank you
The text was updated successfully, but these errors were encountered:
I am using the C-API function ZSTD_CCtx_setPledgedSrcSize() for this purpose
Why are you setting a pledged source size when it might not be accurate? Wouldn't a better solution be to not set the pledged source size, then you can have a frame of whatever size you please?
If you need to know the source size on the decompression side, you could communicate it out-of-band. That would allow you to over/under-estimate because you can have a protocol that adjusts it.
I forgot to report. Setting the pledged size actually works as expected, if you use it correctly. Having blocks with larger pledged size than actual content decompress correctly to the end and then throw an unexpected end of input error (which can be caught and handled). So this can be closed.
I have a scenario where I continuously receive streamed data and I want to compress this data in multiple frames with a pre-given FrameSize. I am using the C-API function ZSTD_CCtx_setPledgedSrcSize() for this purpose. The problem arise when the data stream stops or requests closing the file and starting a new file in a time point when the the current frame size is not yet reached. I can't close the frame in this case. I thought I could do padding with meaningless data till the end of the frame. However, this would not be clean when decompressing the file. If one could use something like skippable blocks, then I could fill the rest of the frame with these and the decompressor would skip them automatically. This could solve my problem. Otherwise I would appreciate any help to overcome this issue.
Thank you
The text was updated successfully, but these errors were encountered: