Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery Storage API integrations should accept either v1beta1 or v1 client #84

Closed
tswast opened this issue Apr 24, 2020 · 3 comments · Fixed by #55
Closed

BigQuery Storage API integrations should accept either v1beta1 or v1 client #84

tswast opened this issue Apr 24, 2020 · 3 comments · Fixed by #55
Assignees
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@tswast
Copy link
Contributor

tswast commented Apr 24, 2020

I notice that the v1beta1 API is still used.

def _download_table_bqstorage_stream(
download_state, bqstorage_client, session, stream, worker_queue, page_to_item
):
position = bigquery_storage_v1beta1.types.StreamPosition(stream=stream)
rowstream = bqstorage_client.read_rows(position).rows(session)
for page in rowstream.pages:
if download_state.done:
return
item = page_to_item(page)
worker_queue.put(item)

This is good, as it keeps packages such as pandas-gbq from breaking, but shouldn't we start thinking about migrating folks to the v1 endpoint? To do that without causing too much disruption, the to_dataframe, magics, and DB-API integrations should accept either v1beta1 or v1 clients for an extended period of time.

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Apr 24, 2020
@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label Apr 27, 2020
@plamut
Copy link
Contributor

plamut commented Apr 28, 2020

@tswast Now that the BQ Storage client is stable, #55 will start using it as a default, and also migrate to v1 stable from v1beta1.

If this would indeed represent to big a disruption, the PR needs to be adjusted to support both Storage client versions. Possibly with a warning if beta client is used?

Also feel free to bring this point up on the linked PR, if you have time, as this feature request directly affects it, thanks!

@plamut plamut added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed triage me I really want to be triaged. labels Apr 28, 2020
@tswast
Copy link
Contributor Author

tswast commented Apr 28, 2020

Yes, that would be a very big disruption. Definitely a breaking change, as it would break pandas-gbq among others.

@plamut
Copy link
Contributor

plamut commented Apr 28, 2020

Noted. We will add this then, and branch on the BQ Storage client type (and if v1beta1 is passed, the old logic for constructing the session will be used).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants