Notice: Change to the email rate limits (still unrestricted for custom SMTP)

[kiwicopple]

10 August 2023 edit: We have introduced a new integration with Resend which makes it incredibly easy to set up SMTP. Read more here

---

Over the past few days we've had a lot of AI applications being built and they are causing an influx of spam messages. We've had to take protective measures to ensure that everyone can continue testing. Because of a few customers treating the email system as a spam system, we had to decrease the number of emails that you can send to 4 per hour. Unfortunately, we are facing a tragedy of the commons, and we had to take this action to protect legitimate users.

The "default" email system we provide is only for testing purposes - it is not intended for production use. This means that the deliverability will not be great, because everyone is using it for testing and is a major reason we don't recommend using it for production.

We're incredibly sorry if this has impacted your project. I would personally go for AWS SES as your custom SMTP provider - it's very cheap and reputable. We also could have been a lot better around the communications of this change, but we were under pressure to make changes (by AWS) otherwise the entire system would have been disabled completely. We're doing a retrospective on how we can better communicate the intent of the email system, and how we can handle urgent changes like this more elegantly.

I want to give a big shout out to all those in the community who helped share the comms and redirect everyone towards using a custom SMTP solution 🙏

[idea-garage]

Thanks for correspond. Stability prioritize all. I relived to hear it doesn't be affected if we're using custom SMTP provider such as SendGrid.

[jesse-stewart]

I attempted to add a custom SMTP, but only received 500 errors. I tried switching back the Supabase SMTP to debug, but the 500 errors persist.

[ashot]

Yes, I can confirm this behavior. If you add a custom SMTP, then switch back to built in supabase smtp, it will no longer work at all with this in the log:
"actor_via_sso":false,"log_type":"user"},"component":"api","error":"Error sending recovery email: 535 Authentication Credentials Invalid","level":"error","method":"POST","msg":"500: Unable to process request","path":"/recover"

[jesse-stewart]

I was able to get sendgrid configured - but it seems like a bug

[kiwicopple]

hit team, can you please reach out to support if this bug persists? It's hard to know if this was an issue on our side or on the SMTP provider's side (or just incorrect SMTP details) without diggin into the details

[ashot]

to be clear the error happens after you switch back – supabase email never works again (not due to limits just all emails fail). something in the code to turn custom smtp off is not clearing state properly.

[GaryAustin1]

https://github.com/supabase/supabase/issues/16150
There is an issue open on this. Once you start with custom SMTP and get an error, you can't currently go back to Supabase SMTP. You have to fix your error or get a working SMTP configuration going. Seems like something in the dashboard getting confused by left over fields.

[jesse-stewart]

I was able to get the SMTP configured correctly and it is no longer an issue. But I think there is a bug where if a user tries setting the SMTP, fails for whatever reason, then sets the SMTP settings back to supbase default, the emails continue to fail

…

On Thu, Jul 27, 2023, 1:16 PM Copple ***@***.***> wrote: hit team, can you please reach out to support if this bug persists? It's hard to know if this was an issue on our side or on the SMTP provider's side (or just incorrect SMTP details) without diggin into the details — Reply to this email directly, view it on GitHub <#15896 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXBOYLSYTA4DPKJKKRXUETXSKPAJANCNFSM6AAAAAA2PXFXPU> . You are receiving this because you commented.Message ID: ***@***.***>

[micheltucker]

Thanks for the update. I have seen a decrease in sign-ups of -27.6% and was wondering why that was the case, but this explains it.

[jesse-stewart]

A quick note to share my experience so far. Switched to sendgrid and tried postmark. Apparently you need to pay for a dedicated IP or your transaction emails with go directly to the spam folder.

I wish Supabase would just offer SMTP for a fee. I can't even turn on email verification because of the spam folder problem.

[Ivan4488]

I second this, I would prefer and happily purchase this directly from Supabase if they offered.

[micheltucker]

A quick note to share my experience so far. Switched to sendgrid and tried postmark. Apparently you need to pay for a dedicated IP or your transaction emails with go directly to the spam folder.

I wish Supabase would just offer SMTP for a fee. I can't even turn on email verification because of the spam folder problem.

Really? I don't seem to have delivery issues and I have been using Sendgrid for Onboarding email thus far.

[justusbluemer]

This feels like a pretty major change that could impact a large number of projects. I would have expected, ironically, an email from Supabase to let me know rather than me accidentally stumbling upon it through some Launch Week announcements involving Resend.

[kiwicopple]

(I'll try to keep the replies consolidated) I have added a response here: https://github.com/orgs/supabase/discussions/15896#discussioncomment-6873221

I agree, we could have done better (and we will in the future)

[JKarlavige]

Just ran into this as well. Also pretty disappointed there was no notification or alerts about this change.

[kiwicopple]

I agree, we could have done better with our communications here. In the future we will:

• Add an incident on status.supabase.com (this was done on the day)
• Send a notification in the dashboard (this was done on the day)
• Update the Dashboard notification system to include email alerts if you want to subscribe to them (this was not done)
• Send a personalized email to anyone we think might be affected

This is no excuse, but in our defense we have always recommended using custom SMTP for production. We were under attack, and we had to make this changes in real time to protect our platform (AWS are extremely strict with SES).

Other changes we have made since this incident:

• We have implemented a "rate limit breached" email notification for the Auth System
• We have built a "one click" integration for providers like Resend to make SMTP even easier
• We have added alerts in the dashboard (although perhaps we need even more copious alerts added more)

[jcuenod]

Can I suggest adding the "4 per hour" limit to this warning?

[kiwicopple]

we're about to put it back up to 30. we'd prefer to let the documentation be the "source of truth" for this, otherwise we need to roll out dashboard changes during an incident which doesn't feel like a best-practice

[jcuenod]

This was very poorly communicated. I am very excited by the work that supabase is doing, but I hope the main thing you guys intend to build is a reputation for stability and customer support. Without those two things, no one can build with you in the long term. This is a change that damaged that reputation in my mind. Still excited, just do better.