Allow users to BYO S3-compatible storage backend

[inian]

We should make it easy for users to export their objects. Also useful for backups etc. We can probably use the S3 replication functionality to do this.

Relevant discussion supabase/supabase#1930

[d-mok]

Is there any update on this? The lack of backup-from-supabase-storage function is stopping me to use supabase storage.

[inian]

Hi @d-mok, we couldn't find an easy way of doing this with the S3 replication functionality above since there there is a limit of 1,000 replication policies per bucket which doesn't work at our scale.

If you have any ideas on how to go about implementing this in a multi-tenant way, let me know!

[ornithophile]

@inian Hi, are there any other plans for syncing files out of Supabase Storage?

[d-mok]

@inian
Then, may be the direction to go is to allow users to provide their own S3 bucket? Like, a bring-your-own-bucket approach?

[ornithophile]

This would be perfect.

…

On Sun, Aug 8, 2021, 6:32 PM d-mok ***@***.***> wrote: @inian <https://github.com/inian> Then, may be the direction to go is to allow users to provide their own S3 bucket? Like, a bring-your-own-bucket approach? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALGWU44AV4UUCIW7UDHU2HTT34V3VANCNFSM46R3X7HA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[thebengeu]

What I'm thinking is to have a new replication section in the UI where you can enter an access key ID and secret access key for an AWS IAM user, and a bucket, and then we'll regularly replicate to that bucket. How does that sound?

[d-mok]

What I'm thinking is to have a new replication section in the UI where you can enter an access key ID and secret access key for an AWS IAM user, and a bucket, and then we'll regularly replicate to that bucket. How does that sound?

So we (users) provide a S3 bucket, and supabase regularly backup everything to that bucket? That sounds good enough.
But I think a better option is to allow user to bring-their-own-bucket. In this way, I can interact with my bucket both through supabase or via some other s3 client. It would be best if, for example, when I add a file to my bucket through another client, supabase can also register the change. This is consistent with the principle how supabase is dealing with Postgresql: we can use the postgresql database any way we want, and supabase will work accordingly.

[tehnorm]

I'd mirror what @d-mok is expressing. Bring your own S3 bucket is very useful for cases that data governance is a consideration as well. Usually for larger organizations have these challenges. It seems lately many SaaS offerings do provide a bring-your-own-bucket model.

[grschafer]

Another vote in favor of bring-your-own-bucket: I think it would allow us to manage data transfer costs better (e.g. keeping transfers within AWS infrastructure in the same region and AZ) which is important for data/processing-heavy applications.

I opened a separate discussion about same-region transfer costs (supabase/supabase#3403) to not distract from the replication focus of this issue, but I wanted to leave a comment here and link the two ideas, because bring-your-own-bucket seems to be a good solution for both.

[the-kro]

I’d also like to see BYOB!

[rlee1990]

Any update on this?

[narve]

Yes, status please? Is it coming (soon), or should we find other workarounds?

[bkuermayr]

would also love to see this

[eqoram]

Any update on this?