PseudoKV roadmap (discussion)

[MeijeSibbel]

Wanted to use this issue to index some further PseudoKV action items we have listed/discussed on discord. Please comment below so we an expand this list.

Items

(No particular order of importance)

• (i) Optimizing parallel transfers with "overdrive," i.e. optionally uploading/downloading more shards than necessary, and discarding the slowest ones. Avoids having to wait for very slow hosts.
• (ii) 1x redundancy prioritization.
• (iii) Sector buffering.
• (iv) LRU cache for downloading recently-uploaded files.
• (v) Context based cancellation.
• (vi) Integrate RHP3.
• (vii) ReadonlyPseudoKV (ephemeral accounts for downloads)
• (viii) Web-socket support for websites.
• (ix) Host cooldown / unreliable host handling (PseudoKV host cooldown #104)
• (x) PseudoKV Garbage collection

[MeijeSibbel]

Some observations/ideas:

• We currently see an average upload speed of 120 Mbps with 5-of-20 using 40 hosts. This is ok but not great. This is where (i/ii) might come in handy so we can bump up the upload speed?
• Junpei and i discussed that it might be a good idea to add a --upload-cache flag to the gateway so that the user can chose to add sector buffering (iii), which will increase storage efficiency but decrease time-to-availability of individual file transfers as it has to wait for other files to reach the packing threshold. This is still most likely the best strategy for uploading lots of small files.