fix(db_api): ensure DDL statements are being executed

[IlyaFaer]

This PR ensures that DDL statements are being executed.

Currently, DDL statements aren't executed until a non-DDL statement is executed. This leads to unexpected behavior. This fix will execute DDL statements immediately when autocommit=True and when commit() is called for autocommit=False.

Note that DDL statements will still be executed when a non-DDL statements is executed in autocommit=False. e.g.

cur.execute(
"""
CREATE TABLE Singers (
    SingerId     INT64 NOT NULL,
    Name    STRING(1024),
) PRIMARY KEY (SingerId)
"""
)

# CREATE TABLE statement will be execute here; before the INSERT statement.

cur.execute(
"""
INSERT INTO Singers (SingerId, Name) VALUES (1, "Name")
"""
)

[larkee]

I think we should execute the statements immediately regardless of whether autocommit is set or not. Because Spanner handles DDL separately from transactions, there's no way to roll back DDL statements (which we will need to clearly document) and so I think having different behavior based on autocommit is unnecessary.

@AVaksman WDYT? Note that we now support executing multiple DDL statements at once which should alleviate the batching concern.

[IlyaFaer]

@larkee, no problem, I'll delete the line.

I'm writing a system test to check the case.

Also I've found out that in !autocommit mode DDLs are also not executed, even if one will call commit(). Will fix it in this PR as well.

[IlyaFaer]

@larkee, you know, about regarding autocommit mode... I think users don't usually go deep into frameworks code. So, we support DDLs separated with ;, still users may use frameworks, which are generating and executing DDLs one by one. In this case it's probably better to look after autocommit mode, to make DDLs pumping up in !autocommit mode - to execute them all at once little bit later.

[larkee]

Hmm, that's a good point. In that case, we'll need to add run_prior_DDL_statements() to commit() as currently the statements aren't being executed even when commit() is called.

[IlyaFaer]

@larkee, something failed in system tests, but it doesn't seem to be related, as DB API system tests live in another file:

__________ TestSessionAPI.test_transaction_batch_update_w_parent_span __________

self = <tests.system.test_system.TestSessionAPI testMethod=test_transaction_batch_update_w_parent_span>

    def test_transaction_batch_update_w_parent_span(self):
        try:
            from opentelemetry import trace
        except ImportError:
            return

        tracer = trace.get_tracer(__name__)

        retry = RetryInstanceState(_has_all_ddl)
        retry(self._db.reload)()

        session = self._db.session()
        session.create()
        self.to_delete.append(session)

        with session.batch() as batch:
            batch.delete(self.TABLE, self.ALL)

        insert_statement = list(self._generate_insert_statements())[0]
        update_statement = (
            "UPDATE contacts SET email = @email " "WHERE contact_id = @contact_id;",
            {"contact_id": 1, "email": "phreddy@example.com"},
            {"contact_id": param_types.INT64, "email": param_types.STRING},
        )
        delete_statement = (
            "DELETE contacts WHERE contact_id = @contact_id;",
            {"contact_id": 1},
            {"contact_id": param_types.INT64},
        )

        def unit_of_work(transaction, self):

            status, row_counts = transaction.batch_update(
                [insert_statement, update_statement, delete_statement]
            )
            self._check_batch_status(status.code)
            self.assertEqual(len(row_counts), 3)
            for row_count in row_counts:
                self.assertEqual(row_count, 1)

        with tracer.start_as_current_span("Test Span"):
            session.run_in_transaction(unit_of_work, self)

        span_list = self.memory_exporter.get_finished_spans()
>       self.assertEqual(len(span_list), 6)
E       AssertionError: 9 != 6

tests/system/test_system.py:1774: AssertionError

[larkee]

This is a known flaky test that I have an issue for: #165

I'll force the tests to rerun 👍

[larkee]

LGTM, just some comments on the system tests

[IlyaFaer]

@larkee, checks passed. They stuck once for 5 hours, so I've reexecuted them