Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reading query table.list_rows based on start_index and max_results not working. #113

Closed
yantrikskaran opened this issue May 19, 2020 · 5 comments
Assignees
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@yantrikskaran
Copy link

yantrikskaran commented May 19, 2020

I am using python google.cloud.bigquery.client library.

Scenario:
I have a query which is supposed to return 2M records. Due to another issue I am facing - #112 (comment)

I thought of using of start_index and max_results on table.list_rows method on temp table of query results.
Rererence -
https://cloud.google.com/bigquery/docs/paging-results#paging_through_query_results

I create a list of start_index and started multiple thread to read 1000000 records from the start_index.

What I notice is that it is fetching same records more than once.

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label May 19, 2020
@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label May 20, 2020
@HemangChothani HemangChothani self-assigned this May 20, 2020
@HemangChothani HemangChothani added type: question Request for information or clarification. Not an issue. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. and removed triage me I really want to be triaged. type: question Request for information or clarification. Not an issue. labels May 20, 2020
@HemangChothani
Copy link
Contributor

@yantrikskaran This is the duplicate issue of #4 and it's already resolved in #27, so please wait for a new release, for now you can verify or use the master branch https://github.com/googleapis/python-bigquery therefore closing this issue.

@yantrikskaran
Copy link
Author

@yantrikskaran This is the duplicate issue of #4 and it's already resolved in #27, so please wait for a new release, for now you can verify or use the master branch https://github.com/googleapis/python-bigquery therefore closing this issue.

@HemangChothani Thanks a lot for pointing out. Any ETA for next release?

@HemangChothani
Copy link
Contributor

@yantrikskaran I am not sure about, @plamut do you have any idea about next release?

@plamut
Copy link
Contributor

plamut commented May 20, 2020

@HemangChothani @yantrikskaran Pushed this forward just yesterday. There are a few things to check (as @shollyman told), but generally speaking there's an agreement that enough fixes have accumulated since the last release and it's time to make a new one.

@yantrikskaran
Copy link
Author

@HemangChothani @yantrikskaran Pushed this forward just yesterday. There are a few things to check (as @shollyman told), but generally speaking there's an agreement that enough fixes have accumulated since the last release and it's time to make a new one.

Looks like we are nearing a release. That s Great News.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

4 participants