Support S3 compatible storage as shared cache

[secustor]

Currently only GCS is supported as shared cache which makes it, inefficient to run ephemeral instances outside of GCP.

Proposal:
Use the Minio SDK to enable to connect to S3 compatible storages. That way users can use other Backends too which are supported by Minio.

[Swatinem]

What exactly is your use case like? I’m wondering if the shared cache would even make as much sense for situations outside of Sentrys own scale.

[secustor]

We are planning to host sentry components in a stateless cluster in which the nodes and therefore symbolicator instances will be possibly recreated multiple times per hour.
My understanding is that symbolicator instances are downloading the sources from the sentry filestore via the web instances and therefore create significant load on the web instances.
I would like to prevent that using the shared cache.

[Swatinem]

Ah yes, in that case if you have an autoscaling cluster that is creating tons of nodes on the fly thats the perfect usecase for this indeed.

We already have the S3 SDK integrated for downloads from external symbol sources, maybe doing something similar for the shared cache is not too much work.

I did some refactors in the shared cache recently, but things should be more quite around that code right now. Feel free to open up a PR. I doubt our team itself will prioritize this anytime soon as we are using GCS for our own SaaS and single tenant deployments.