RFC: Per-client media selective sync settings

[gmaclennan]

Description

Our Co-design team has explored UX for sync that allows a user to choose whether to upload/download just thumbnail and preview images, or full-size images. When internet sync is enabled, a user might want to sync full-sized images with local clients, but only thumbnail images with remote (internet-connected) clients.

This is difficult to implement due to the way Hypercore works: choosing what to download is done on the hypercore instance, and applies across all peer connections, and internally Hypercore will choose the most efficient way to download the selected blocks from all connected peers. There is no way exposed in the API to select which blocks should be downloaded based on which peer they are coming from: if the peer has it, then it could be downloaded.

Solution 1: Hypercore Sessions

We could use hypercore sessions to create a unique session for each media sync option: thumbnails; thumbnails + previews; all media including originals. For each group of hypercore sessions we could then configure the LiveDownloader to download the selected media, and then selectively replicate these sessions based on client options.

E.g. Peer A connects to Peer B. Peer B sends an extension message to A saying "please only download thumbnails from me". Peer A would then create separate sessions for the media hypercores, and create a liveDownload instance with those sessions that only downloads media with the thumbnail prefix. Peer A would then add those sessions to the replication stream with peer B, rather than the main session which would be downloading all media data.

Solution 2: Protocol Hacks

Solution 1 (above) relies on trust between peers: there is no way in Hypercore currently to control what a connected peer can download from you, so solution 1 relies on a request "please only download this from me", and trust that the connected peer will respect that request.

An alternative solution would be to add additional functionality to the Hypercore replicator, so that when a peer receives a "want" message, it selectively responds with "have" messages only for blocks that have been enabled for sharing. In addition, when a peer responds to a "request" message, it would ignore it for any blocks which have not been enabled for sharing.

This solution requires changes to the Hypercore code, and would need to be implemented either as a patch, or an upstream PR to Hypercore.

Suggested Approach

Solution 1 is simpler to implement in the short-term, and will be easier to maintain than a patch, which could break when hypercore code changes. This selective sync is not used for security (e.g. blocking users from syncing certain blocks), so relying on trust is probably ok - as long as everyone is using a Mapeo app then it will respect these requests to only download certain data.

Solving this is particularly relevant for cloud sync, because peers on a slow or expensive internet connection might only want to "upload" (e.g. for the cloud to download) thumbnails of images, where-as peers with a fast connection might want to send all images (including originals) to the cloud server. Currently the cloud server would need to be configured with what to sync and it would apply to all peers.