Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support stale reads #146

Merged
merged 6 commits into from
Nov 19, 2021
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,14 @@ Note that execution options are applied lazily - on the `execute()` method call,

ReadOnly/ReadWrite mode of a connection can't be changed while a transaction is in progress - first you must commit or rollback it.

### Stale reads
To use the Spanner [Stale Reads](https://cloud.google.com/spanner/docs/reads#perform-stale-read) with SQLAlchemy you can tweak the connection execution options with a wanted staleness value. For example:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read timestamp and minimum read timestamp expect a datetime.datetime object.
Maximum staleness and exact staleness expect a datetime.timedelta object.
This should probably be documented here in some way, possibly with examples for each, WDYT?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, added examples for all of them

```python
with engine.connect().execution_options(staleness={"max_staleness": 5}) as connection:
IlyaFaer marked this conversation as resolved.
Show resolved Hide resolved
connection.execute(select(["*"], from_obj=table)).fetchall()
```
Note that the set option will be dropped when the connection is returned back to the pool.

### DDL and transactions
DDL statements are executed outside the regular transactions mechanism, which means DDL statements will not be rolled back on normal transaction rollback.

Expand Down
13 changes: 13 additions & 0 deletions google/cloud/sqlalchemy_spanner/sqlalchemy_spanner.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@
from sqlalchemy import ForeignKeyConstraint, types, util
from sqlalchemy.engine.base import Engine
from sqlalchemy.engine.default import DefaultDialect, DefaultExecutionContext
from sqlalchemy.event import listens_for
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.pool import Pool
from sqlalchemy.sql.compiler import (
selectable,
DDLCompiler,
Expand All @@ -38,6 +40,13 @@
from google.cloud import spanner_dbapi
from google.cloud.sqlalchemy_spanner._opentelemetry_tracing import trace_call


@listens_for(Pool, "reset")
def reset_connection(dbapi_conn, connection_record):
"""An event of returning a connection back to a pool."""
dbapi_conn.connection.staleness = None
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the docs, connection the hook to the Pool class will connect it to all the pools used by the dialect. In the test below a QueuedPool is used (by default), and it's working fine.



# Spanner-to-SQLAlchemy types map
_type_map = {
"BOOL": types.Boolean,
Expand Down Expand Up @@ -128,6 +137,10 @@ def pre_exec(self):
if read_only is not None:
self._dbapi_connection.connection.read_only = read_only

staleness = self.execution_options.get("staleness", None)
if staleness is not None:
self._dbapi_connection.connection.staleness = staleness


class SpannerIdentifierPreparer(IdentifierPreparer):
"""Identifiers compiler.
Expand Down
12 changes: 11 additions & 1 deletion test/test_suite.py
Original file line number Diff line number Diff line change
Expand Up @@ -1578,7 +1578,7 @@ class ExecutionOptionsTest(fixtures.TestBase):
"""

def setUp(self):
self._engine = create_engine(get_db_url())
self._engine = create_engine(get_db_url(), pool_size=1)
self._metadata = MetaData(bind=self._engine)

self._table = Table(
Expand All @@ -1594,3 +1594,13 @@ def test_read_only(self):
with self._engine.connect().execution_options(read_only=True) as connection:
connection.execute(select(["*"], from_obj=self._table)).fetchall()
assert connection.connection.read_only is True

def test_staleness(self):
with self._engine.connect().execution_options(
staleness={"max_staleness": 5}
IlyaFaer marked this conversation as resolved.
Show resolved Hide resolved
) as connection:
connection.execute(select(["*"], from_obj=self._table)).fetchall()
assert connection.connection.staleness == {"max_staleness": 5}
IlyaFaer marked this conversation as resolved.
Show resolved Hide resolved

with self._engine.connect() as connection:
assert connection.connection.staleness is None
IlyaFaer marked this conversation as resolved.
Show resolved Hide resolved