BufferedReader/writer for general IO?

[grero]

I was wondering whether it would be feasible to have BufferedReader/Writer work for general io, rather than just for IOStream? The reason I'm asking is that I'm experimenting with reading parts of large jld2 files (several 10s of G) over sftp. I implemented an IO compliant interface for reading/writing and seeking, and now I would like to have BufferedReader be able to read from this IO. Looking at the source, it seems that if the io field of BufferedReader were of type IO rather than IOStream, I could use it without having to re-implement the BufferedReader interface. I'd be interesting in any thoughts on this approach. Thanks!

[JonasIsensee]

Hi @grero ,

this sounds like an interesting idea that could be useful in some applications.
I must admit I have not looked at that part of the code before and at this point cannot help with that
but your suggestion sounds plausible.

I would recommend that you fork JLD2 and just try it out
and you are very welcome to report your findings.
If it works and doesn't break anything else in JLD2, we could then likely adapt the signatures to your needs.

[grero]

I'm close to having something that sort of works here: grero/SecureRemoteFiles.jl#4
Once I have something a bit more polished perhaps we can discuss whether to include the functionality in JLD2 itself. Thanks for your reply!

[JonasIsensee]

Hi @grero,

I just wanted to ask whether you've made any progress on this

[grero]

Hey,
I'm so sorry, I sort of got distracted with other projects. Regarding the tests, I was trying to figure out how best to structure them. I gather there aren't currently any tests that explicitly tests the BufferedReader? Would the best approach be to add those, or to basically run the current tests again, using a mocked up bytesavailble?

[JonasIsensee]

Hi,
no worries. This isn't urgent but I just didn't want the idea to be forgotten.

I think you're right. I haven't seen any explicit tests for that either.
If you've had a look at the tests you'll realize that they are in a bad state in terms of organization.
Anything is appreciated but I think it'll be best to just add a few small tests for your mockup bytesavailable.
(Really just some tiny tests that reproduce the bug you found)