Streaming mode: Support "external dictionary" accessed via callback (in addition to "dictionary buffer in RAM")

[mikevoyt]

Hi there - if you are interested, here are some small changes to your library, which allow for reading the compressed input, and writing the decompressed output, a single byte at a time. I've also included a new test program to demonstrate the changes in uncompressed_streaming.

With these changes, you can open an arbitrary sized compressed file and inflate it, without calling malloc for either the input or the output. I'm using the changes in this fork to decompress images on an embedded system from one location in flash memory to another (by replacing readDest and readSource to read a single byte from flash memory).

I've also added a simple shell script for testing, by using /dev/urandom and gzip for creating random input files, then inflating them and comparing the difference.

Just thought I'd share these changes if you are interested in merging back into master - let me know if you'd have any questions or would like to see some changes.

[pfalcon]

Thanks, but I don't think the description of your patch is correct. "Streamed io" is what I implemented in v2.0. It works either for single byte reads from compressed stream, or for multiple bytes.

What your patch appears to do is to deflate a stream without memory buffer, by using backing store as such buffer, by trading fixed-size memory buffer for potentially unlimited-size backing store. What's usecase for that?

Tests are definitely needed (as it stands now, MicroPython is used for testing uzlib, it would be nice to have some tests included in uzlib distribution), but they should be separated from other changes. So, feel free to submit these as different pull requests.

[mikevoyt]

Yes, you are correct - the intention of this patch is to deflate a stream without a memory buffer. By "streamed-io", I was referring to streaming from the backing store during the decompression, rather than holding the compressed and/or decompressed data in memory.

Here's my use case: I'm using this for remote software update of an embedded device (no malloc, 32K of RAM, and a few MB of flash storage).

I can compress the software update image using gzip on a host computer, and stream it to the embedded device, which will store the compressed image in an unused bank of flash memory.

After writing the image to flash and successfully verifying the checksum, I then decompress the image, writing it directly into the executable region of flash memory.

In this way, the executable image can be several hundred KB or more; and yet, it requires very little RAM to decompress. On such a device, it is not possible to contain either the entire compressed or entire uncompressed image in RAM.

[MrSapps]

I'm confused.. so does tinf supported streamed decompression without this PR already?