Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream BufRead decompressor does not consume all the compressed bytes #255

Open
Ten0 opened this issue Dec 18, 2023 · 0 comments
Open

Stream BufRead decompressor does not consume all the compressed bytes #255

Ten0 opened this issue Dec 18, 2023 · 0 comments

Comments

@Ten0
Copy link

Ten0 commented Dec 18, 2023

Stream BufRead decompressor does not consume all the compressed bytes, despite reading all the decompressed bytes and calling finish().
This is an issue when doing block compression/decompression.
No other decompressor I have tested seems to have this behavior (deflate with flate2, bzip2 with bzip2, xz with xz2).

Reproduction

The last assert fails:

fn main() {
	let to_compress = &[54, 6, 102, 111, 111, 84, 6, 98, 97, 114];
	let mut encoder = zstd::stream::write::Encoder::new(Vec::new(), 0).unwrap();
	std::io::Write::write_all(&mut encoder, to_compress).unwrap();
	let buf = encoder.finish().unwrap();
	assert_eq!(
		buf,
		&[40, 181, 47, 253, 0, 88, 81, 0, 0, 54, 6, 102, 111, 111, 84, 6, 98, 97, 114]
	);

	let mut decoder = zstd::stream::read::Decoder::with_buffer(buf.as_slice()).unwrap();
	let mut decompressed = Vec::new();
	let mut buf = [0u8; 1];
	for _ in 0..to_compress.len() {
		assert_eq!(std::io::Read::read(&mut decoder, &mut buf).unwrap(), 1);
		decompressed.extend_from_slice(&buf);
	}
	assert_eq!(decompressed, to_compress);

	let reader_after_decompression = decoder.finish();
	dbg!(&reader_after_decompression);
	assert_eq!(reader_after_decompression, &[]); // Panics: `reader_after_decompression` is `&[114]` now.
}

Meaning that the reader hasn't actually consumed all the bytes of the compressed input, despite all decompressed output being properly decompressed and read and finish being called.

Note that this is specific to the scenario where it's the reader stopping to ask for bytes.

(zstd 0.13)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant