Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job#getQueryResultsStream fails when passed a startIndex #1099

Closed
felix-d opened this issue Apr 13, 2022 · 3 comments · Fixed by #1100 · May be fixed by #1178
Closed

Job#getQueryResultsStream fails when passed a startIndex #1099

felix-d opened this issue Apr 13, 2022 · 3 comments · Fixed by #1100 · May be fixed by #1178
Assignees
Labels
api: bigquery Issues related to the googleapis/nodejs-bigquery API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@felix-d
Copy link
Contributor

felix-d commented Apr 13, 2022

Environment details

  • OS:
  • Node.js version:
  • npm version:
  • @google-cloud/bigquery version: latest

Steps to reproduce

  1. Pass startIndex to Job#getQueryResultsStream
  2. First request to populate the stream is successful and starts at the correct index.
  3. Second request fails with When using a page token, you cannot specify an arbitrary startIndex

It seems this is simply because startIndex should be used on the first fetch, and then discarded once page tokens can be used for the next query.

@felix-d felix-d added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Apr 13, 2022
@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/nodejs-bigquery API. label Apr 13, 2022
@steffnay
Copy link
Contributor

Can you provide the code you're running that produces this error?

@felix-d
Copy link
Contributor Author

felix-d commented Apr 18, 2022

Sure! You can reproduce by simply creating a BigQuery job stream with startIndex. It will work for the first request and start at the right startIndex, then throw if there's a second page to be fetched in the result set because the library also forwards pageToken in addition to startIndex and the server doesn't accept both (which makes sense).

const job = new BigQuery(...).createJob(...);
const stream = job.getQueryResultsStream({ startIndex: 2000 });

for await (const row of stream) { // throws on the second page
  // ...
}

Since the library forwards the pageToken cursor parameter on the second request, startIndex becomes obsolete. Therefore, if we use a pageToken from a server response, we can simply discard startIndex.

@felix-d
Copy link
Contributor Author

felix-d commented Apr 29, 2022

Hey @steffnay, any progress on this? We currently have to run a fork and would like to get back on the bandwagon at some point :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/nodejs-bigquery API. priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
2 participants