-
Notifications
You must be signed in to change notification settings - Fork 115
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"This slot has been invalidated because it exceeded the maximum reserved size" error #1089
Comments
This appears to have been caused by me accidentally sending a very large number of SQL queries to the Electric migration proxy rather than to my Postgres database (where I intended to send them). Easy mistake to make... might be worth handling this case more gracefully. |
@djbutler How did you get things back to normal? |
I ran the SQL given on the “Roadmap” page for deleting the database
structures created by electric. When you start the electric service again,
it will re-created the needed structures. The errors stopped. But
unfortunately, when I re-electrified my tables, the data already in the
tables does not sync, so that’s the next thing to debug. New data added to
the tables syncs fine.
…On Wed, Mar 27, 2024 at 11:54 PM 승우 ***@***.***> wrote:
@djbutler <https://github.com/djbutler> How did you get things back to
normal?
—
Reply to this email directly, view it on GitHub
<#1089 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AABKG4KRPM6V4V5HZIPAXMLY2O5CVAVCNFSM6AAAAABFE3ZROCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRUGUZDGMZXGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Also, if I didn’t mention it earlier, the root cause seems to be that I
accidentally issued a ton of data-intensive queries to the migration proxy
that were in fact intended for the underlying PSQL database. That messed
up electric.
…On Thu, Mar 28, 2024 at 9:07 AM Dan Butler ***@***.***> wrote:
I ran the SQL given on the “Roadmap” page for deleting the database
structures created by electric. When you start the electric service again,
it will re-created the needed structures. The errors stopped. But
unfortunately, when I re-electrified my tables, the data already in the
tables does not sync, so that’s the next thing to debug. New data added to
the tables syncs fine.
On Wed, Mar 27, 2024 at 11:54 PM 승우 ***@***.***> wrote:
> @djbutler <https://github.com/djbutler> How did you get things back to
> normal?
>
> —
> Reply to this email directly, view it on GitHub
> <#1089 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABKG4KRPM6V4V5HZIPAXMLY2O5CVAVCNFSM6AAAAABFE3ZROCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRUGUZDGMZXGU>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
Hey @djbutler. Thanks for reporting this! By default, Postgres does not put a limit on replication slot sizes. Are you using a hosted database or do you set the relevant configuration option in your Postgres config? On the issue of disk usage growth caused by the replication slot Electric creates: this is, in fact, a hairy problem. You'll find more relevant info here - #1083. TL;DR, any write to PG will cause Postgres disk usage to grow, even if the write itself does not touch electrified tables. In the next release of Electric, there will be a new configuration option to limit the maximum size of disk space Electric's replication slot can retain. |
Thanks for the explanation. Yeah this would be great to fix, since it's come up a few times for me. This db is self-hosted on EC2 - it's a Supabase docker image. I don't know if any of these look relevant:
|
@djbutler Thanks for sharing your config! The
Even then, I guess once a replication slot has been invalidated, it can no longer be used. You can drop Electric's slot with
In the upcoming release of Electric we're introducing a configuration parameter to set the upper limit on WAL size that Electric can retain, see the preview here. |
Thanks @alco, I appreciate the detailed explanation. One quick question about the preview you linked to - it says:
So if I understand correctly, the local copy will be discarded because the "diff" information between the client's copy and the server's newer copy has been thrown away? Also, discarding the local copy of the server state doesn't cause pending client-side transactions to be dropped does it? That would be pretty bad - it could lead to data loss. I assume it's just a performance issue - the client will have to download the server state again, but won't lose pending local transactions, right? |
Electric does not compute diffs currently. It can stream transactions from Postgres' WAL and, provided there's no gap between what it has already sent to a given client and what it can stream from Postgres, it will keep streaming new transactions to the client. But if the replication slot is removed, Postgres will discard old WAL segments and so, upon creating a new replication slot, Electric will only be able to resuming streaming from the latest state in Postgres. If it detects that there's a gap between the last transaction the client received and the earliest transaction Electric can stream from Postgres, it will ask the client to reset its local state to force it to request the latest snapshot of the shape data from the database before going back to the transaction streaming mode.
The idea is that client's local updates will survive the reset. This is a gray area currently since we don't yet have sufficient test coverage for this edge case but we'll get there eventually. |
Okay got it, thanks! |
My electric service crashed, and when I try to restart it with docker compose, I get this:
The text was updated successfully, but these errors were encountered: