Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question - how to configure Tortoise for transparent auto-reconnect? #1604

Open
davidmcnabnz opened this issue May 7, 2024 · 2 comments
Open

Comments

@davidmcnabnz
Copy link

Is your feature request related to a problem? Please describe.
PostgreSQL suffered a segfault of a worker process, causing my program using Tortoise to suffer exceptions when attempting to access data.

I couldn't find anything in the documentation about how to set Tortoise up to retry lost connections.

Describe the solution you'd like
A set of options I can pass to Tortoise.init() to govern automatic reconnection behaviour.

Also, while reconnection is in progress, for any tasks with current transaction calls to accept keywords indicating whether they want an exception in event of connection entering "retry" state, or want to sleep until connection is re-established.

Describe alternatives you've considered
Presently, my only straightforward option is that when a process notices that Tortoise can no longer talk to the back end database, to have the process self-terminate, so that when systemd re-launches it, it makes a fresh connection attempt with fresh state.

Additional context
Any positive advice appreciated.

@abondar
Copy link
Member

abondar commented May 7, 2024

Tortoise does not have explicit support for connection retries, we had something like that long ago, but scraped it in favour of pooling used in db clients
I am not sure what would be right course of action here, as we expect that pools sort that themselves

You can share your configuration and examples of errors, may be I will be able to look into it, but not sure if we will change something

@davidmcnabnz
Copy link
Author

I'm now configuring Tortoise with a higher maximum pool size.

@abondar or anyone else, can you recommend best practice in cases where a Tortoise-based program suffers an unexpected shutdown of the underlying PostgreSQL database it's connected to?

What I'm currently doing is catching ConnectedRefusedError when attempting transactions. When this occurs, I'm rolling back partially completed processing, then terminating the process, to let systemd start it up again. On restart, my process sits in a loop repeatedly calling Tortoise.generate_schemas() then sleeping until this call succeeds.

Surely there's a better way. Keen for your suggestions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants