URL seeding (BEP-19) and S3 presigned URL's

[siepkes]

This is not really an issue, it's more me checking if I'm not going to create a useless PR ;-).

Currently a web-seed-entry offers functionality to use HTTP basic auth (via the auth field) and supports setting custom headers (via extra_headers).

I'm currently exploring if it would be possible to use S3 pre-signed URL's for URL seeding. A S3 pre-signed URL is basically just a bunch of GET parameters in a URL. However (to my knowledge) there is currently no way of telling libtorrent to add GET parameters to requests it makes for HTTP seeding.

I was thinking about adding something like this to the web_seed_struct:

using get_args_t = std::vector<std::pair<std::string, std::string>>;
[....]

// Any extra GET arguments that need to be passed to the web seed
get_args_t extra_get_args;

Is this something you folks would be open for in a PR?

[arvidn]

It seems somewhat reasonable. However, I assume there's not really any privacy expectation when authenticating this way, is that right?

It is possible to expose S3 files as just regular http requests too, right? Where you wouldn't need the request parameters.

Is there a strong use case behind using pre-signed URLs?

[siepkes]

However, I assume there's not really any privacy expectation when authenticating this way, is that right?

Maybe I don't fully understand what your saying, so if I misinterpreted it, please let me know!

Overall one can have a pretty reasonable expectancy of privacy I think. The query string parameters when doing a GET operation are encrypted as well with HTTPS.

So if someone were to listen in on the connection all they would see is HTTPS "communication" to a specific domain. But for example which bucket, requested offsets, S3 authentication token, etc. (i.e. all the things in the query parameters) are all invisible to someone whose listening in.

The only privacy foot-gun in case of S3 would be that S3 also has a mode where you can specify the bucket name in DNS (something like bucket-foo.s3.aws.com) instead of specifying the bucket in the query parameters. In that case someone listening in on the connection could see the name of the bucket but all query parameters are still not visible. So even that would be of limited use I think.

Is there a strong use case behind using pre-signed URLs?

We use SSLTorrents to distribute private data. However we want to be able to sometimes use HTTP seeding (hosted by S3). For example to speed up transfers. However the data is still considered private. Allowing anyone access to these files hosted on S3 is not preferable. Even though people would still need to obtain the URL we would have no mechanism to block unauthorized access if the URL every leaks. Also in our case the S3 pre-signed seeding URL's would be generated automatically for each peer with a limited life time.

[arvidn]

I see. And the torrent file itself is also considered secret then.

Makes sense. IIRC, the torrent file isn't considered secret in normal SSL torrents, you add a certificate that you authenticate with.

[siepkes]

I see. And the torrent file itself is also considered secret then.

Yeah, in the same sense that you would probably consider the listing of files in your Google Drive or Dropbox to be private. In our case it wouldn't be the end of the world if a Torrent file became public. But we don't have the torrents publicly available.

Just as some additional background; We generate short-lived (week or so) certificates for all peers and for each torrent (which we automatically refresh and manage). This works by the peer generating a private key locally and then contacting an API at our central server and submitting a certificate signing request (CSR) for a specific torrent (for the peers private key). The server then responds with a certificate for that particular torrent and peer.

If a peer is no longer allowed to interact with a torrent (or the peer is deactivated in it's entirety) the central server no longer accepts CSR's for that peer/torrent. Ideally we would implement CRL or OCSP support in libtorrent for SSLTorrent's but for now that seems over the top for our use-case.

To circle back to my initial question; Is a PR providing the ability to supply additional query parameters in the requests of HTTP seeds something you would be open to? Aside from pre-signed S3 URL's I think it might also be useful for other use-cases (just like one can now already set additional HTTP headers with HTTP seed requests).

[arvidn]

Yes, a PR to do that would be welcome