Lift the max object size for non-R2 stores

[wjones127]

Right now, we use ObjectStore::put_multipart() which uses fixed 10MB chunks to write to object stores. Object stores limits part size of 10,000 parts, so this means we have a 100GB object size limit. For some use cases this is too small.

There's two ways we could solve this:

1. Make the part size configurable. This would work, but it would require manual configuration provided by the user, which is less than ideal.
2. Make the part size variable, and increasing linearly. This is the algorithm PyArrow used to use, and it would allow much larger part sizes with no special configuration. The only downside is that R2 would not work with this, as it requires exactly equal part sizes.

IMO the best solution would be 2, but with a carve out for R2 (which will used fixed-size uploads). We already want to have carveouts for R2 and Minio for commit mechanism (see #2246).

In order to implement this, we'll need to use the low-level MultiPartStore API. We'll need to upgrade to a similar API anyways in object_store 0.10.0, since the return value for put_multipart has changed. ¹

Footnotes

1. https://docs.rs/object_store/latest/object_store/trait.MultipartUpload.html ↩

[wjones127]

Possible follow up, when we switch APIs, we'll have more control over retries for uploading a part. Maybe we should retry the Connection reset by peer with increased backoff?