New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
spanner: Client library sending a high rate of BeginTransaction requests when database is deleted #16
spanner: Client library sending a high rate of BeginTransaction requests when database is deleted #16
Comments
This is caused by an infinite loop that can occur during preparing sessions in the session pool. The session pool will try to prepare a session with a read/write transaction when a session is released into the pool. This process follows this logic:
|
@olavloite Thanks for fixing this! I cherry-picked googleapis/google-cloud-java@7c66ee4, and it seems to have fixed the issue for me. |
@skuruppu options:
|
@skuruppu @snehashah16 |
My take on this is that since the Java client connection is at a DB level, if the DB is deleted, then this invalidates the current connection. Essentially option 2 that @snehashah16 mentioned so that the customer is expected to create a new client if they recreate a DB with the same name. I'm a bit weary of sending RPCs continuously until NOT_FOUND errors are no longer returned. In most cases, I suspect once a DB is deleted, users don't recreate the same DB shortly after in a production app. They probably do such things in tests but for that we would expect them to rerun the tests with a new connection. Let me know if I misunderstood something here. |
DatabaseClients should not continue to try to send RPCs to a database that has been deleted. Instead, the session pool will keep track of whether a database not found error has been returned for a database, and if so, will invalidate itself. All subsequent calls for this database will return a DatabaseNotFoundException without calling a RPC. If a database is re-created, the user must create a new DatabaseClient with a new session pool in order to resume usage of the database. Fixes #16
DatabaseClients should not continue to try to send RPCs to a database that has been deleted. Instead, the session pool will keep track of whether a database not found error has been returned for a database, and if so, will invalidate itself. All subsequent calls for this database will return a DatabaseNotFoundException without calling a RPC. If a database is re-created, the user must create a new DatabaseClient with a new session pool in order to resume usage of the database. Fixes #16
DatabaseClients should not continue to try to send RPCs to a database that has been deleted. Instead, the session pool will keep track of whether a database not found error has been returned for a database, and if so, will invalidate itself. All subsequent calls for this database will return a DatabaseNotFoundException without calling a RPC. If a database is re-created, the user must create a new DatabaseClient with a new session pool in order to resume usage of the database. Fixes #16
* fix: stop sending rpcs on deleted db * fix: client should stop sending rpcs after database dropped DatabaseClients should not continue to try to send RPCs to a database that has been deleted. Instead, the session pool will keep track of whether a database not found error has been returned for a database, and if so, will invalidate itself. All subsequent calls for this database will return a DatabaseNotFoundException without calling a RPC. If a database is re-created, the user must create a new DatabaseClient with a new session pool in order to resume usage of the database. Fixes #16 * fix: remove double check on isValid * fix: add wait to deleted db integration test * fix: process review comments * fix: update copyright year
Once a database is deleted and if the Java Spanner client is still left open, it starts sending a large number of BeginTransaction requests. The additional requests fail with status NOT_FOUND.
The following error just keeps repeating in the logs:
I'm guessing this is happening in an attempt to replenish the pool.
The text was updated successfully, but these errors were encountered: