Support of Storage API

[yan-hic]

Any plan on supporting the bigquery storage api ?

---

Edit Sub-steps:

• Update BQ DB-API to support BigQuery Storage API [BigQuery, BigQuery Storage]: Add option to use BigQuery Storage API to download results in BigQuery DB-API python-bigquery#16
• Update create_connect_args to (optionally) return a BQ Storage API client alongside the BQ client.
• Add a use_bqstorage_api option to the connection URL parsing logic.

[tswast]

This requires adding support for the BigQuery Storage API to the DB-API adapter in google-cloud-bigquery, first.

I assume you want to improve the performance of downloading large results?

[yan-hic]

You are right on with your comment in https://github.com/googleapis/google-cloud-python/issues/9465, it is indeed to test/use with superset.

As a workaround, I can try pyodbc with the BQ ODBC driver.

[yan-hic]

@tswast does this close it ? : googleapis/python-bigquery#36 (comment)
If so, some documentation would be useful. Thanks

[tswast]

It unblocks this feature from being completed, but doesn't quite fix it yet.

We need to update create_connect_args to (optionally) return a BQ Storage API client alongside the BQ client.

https://github.com/mxmzdlv/pybigquery/blob/f427dbe0040cbdf3681c1937699af0bea2b89af6/pybigquery/sqlalchemy_bigquery.py#L318

As I understand it, the return value of create_connect_args is passed to the BigQuery DB-API connect function.

This will also require adding a use_bqstorage_api option to the connection URL parsing logic https://github.com/mxmzdlv/pybigquery/blob/master/pybigquery/parse_url.py

[alonme]

Can you guys add some more details about the usage of this feature?
I would like to implement it, but i feel like i won't be able to do it in the best way possible and test that it works without understanding this issue a bit more

[yan-hic]

@alonme nothing to document since not implemented yet.
In the meantime you can use the standard way to read rows, with hope for the turbo mode soon.

[alonme]

Maybe I wasn't clear,
I would be happy to contribute this feature.
On order to do that, id like to understand it better

[tswast]

@alonme Thank you for volunteering to help.

To verify the updates suggested in #41 (comment) make a query to table where the expected result set is moderately large (100 MB or more). When use_bqstorage_api=True is included in the connection string, you should notice a considerable (4-16x or more) speedup.

[alonme]

@tswast cool, got a PoC to work.
hope to get a nice branch for PR in the next days,
please assign to me

[yan-hic]

closing as per #61 (comment)

[gallodustin]

Add a use_bqstorage_api option to the connection URL parsing logic.

Are there any plans to implement this feature as optional? My understanding is that using the storage API incurs an extra financial cost, so it would be nice to be able to disable it when desired.