Unusual behaviour when downloading partial chunks

[traktion]

I've enabled streaming in my sn_httpd client using the files API, but I uncovered some unusual behaviour.

The associated code is here: https://github.com/traktion/sn_httpd/blob/163e59181502e6783b71027ed55328ad7f5920f1/src/main.rs#L170

When using STREAM_CHUNK_SIZE smaller than a safe net chunk size (up to 512 KB, I believe), there seems to be unexpected overlapping data returned.

For example, when using 131072 as the STREAM_CHUNK_SIZE, one of my test images was being corrupted. On closer inspection with Wireshark, it looked like two bytes were being repeated on the subsequent call.

I took the output from Wireshark and compared it. I've attached a screenshot of the difference:

When using much smaller STREAM_CHUNK_SIZE, it exacerbates the issue (although performance suffers regardless).

I can't see any issues in my code and it looks like position is being measured / applied incorrectly somewhere within this code block (inc. dependencies):

safe_network/sn_client/src/files/download.rs

Line 156 in e92b8e4

if let Ok(data_map) = self.unpack_chunk(chunk.clone()).await {

I haven't seen any issues when downloading with position == 0 and MAX length, but for very large files, it's inefficient for streaming using that technique.

[JasonPaulGithub]

Amazing work @traktion
I will cc @RolandSherwin about this and mark the issue as a bug for now. Thanks!

[traktion]

From digging more into this, I tracked down the issue to self_encryption and I have a fix for it. I'll create a PR on that repo instead and we can close this one out if/when it's confirmed.