refactor: replaces job#getQueryResults with table#getRows in query method

[steffnay]

Fixes #12

• Tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

[codecov]

Codecov Report

Merging #454 into master will decrease coverage by 1.68%.
The diff coverage is 8.33%.

@@            Coverage Diff             @@
##           master     #454      +/-   ##
==========================================
- Coverage   99.42%   97.74%   -1.69%     
==========================================
  Files           5        5              
  Lines         699      708       +9     
  Branches      194      196       +2     
==========================================
- Hits          695      692       -3     
- Misses          2       12      +10     
- Partials        2        4       +2

Impacted Files	Coverage Δ
src/index.ts	94.18% <8.33%> (-4.32%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 89f58f8...3fc5369. Read the comment docs.

[codecov]

Codecov Report

Merging #454 into master will increase coverage by 0.14%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #454      +/-   ##
==========================================
+ Coverage   99.42%   99.57%   +0.14%     
==========================================
  Files           5        5              
  Lines         698      708      +10     
  Branches      192      193       +1     
==========================================
+ Hits          694      705      +11     
+ Misses          3        1       -2     
- Partials        1        2       +1

Impacted Files	Coverage Δ
src/index.ts	98.91% <100%> (+0.41%)	⬆️
src/table.ts	100% <0%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7db57d2...6fc30ea. Read the comment docs.

[bcoe]

Hey @steffnay, just came across this while performing triage (I'm not a subject expert in bigquery, so please take my comments with a grain of salt 😛).

This is looking reasonable to me.

[bcoe]

I would pull this regex into a constant outside of the method, similar to the approach used in tests, e.g.,

const MODEL_QUERY_STRING = '\\b((create|replace) model)\\b', 'i'

I don't know this problem space too well, but I wonder if it might be worth tightening up the regex a bit. What would happen if someone performed a query asking with a variable create rather than a statement, e.g.,

SELECT * FROM foo WHERE operation = 'create'

[bcoe]

It surprised me that QueryRowsCallback had to change RowsCallback, since in the case of model queries we still return the original response:

 job!.getQueryResults(options, callback as QueryRowsCallback);

☝️ this is the original slow path right? which we returned as a QueryRowsCallback?

Mainly just making sure code-paths return the same shaped object.

[callmehiphop]

So getQueryResults is going to get called continuously until the job is done and all the rows have been fetched. That doesn't sound like something we want here, so we'll probably need to set autoPaginate to false.

[callmehiphop]

I think I pieced this together, I have a feeling we are going to need to manually poll this method until we see the jobComplete value set to true. If there are 0 rows, we execute the users callback with an empty array and the response object. If a page was returned, then we should call listTableData.

@shollyman does that sound right?

[shollyman]

Yeah. Typical flow for query processing should look something like this in terms of backend interactions:

1. Call bigquery.jobs.insert() to insert the query job into the system,
   which returns a job resource (and job reference) for future polling
   operations.

2. Call bigquery.jobs.getQueryResults to poll (possibly repeatedly) for
   job completion. Typically you do this with a nonzero timeoutMs param (which
   moves polling to the server side as a hanging get), and zero maxRows
   param (as we're not consuming row data here).

3. When job is complete, you can consume schema from the getQueryResults,
   and then setup the node equivalent of a row iterator if the getQueryResults
   response indicates rows were returned. Many queries don't return results,
   such as DML or DDL operations (UPDATE TABLE, ALTER TABLE, etc).

4. Row iteration should use bigquery.tabledata.list to fetch pages of row data.
   Each page optionally returns a next page token, which is passed to the next
   invocation to advance the cursor (don't use explicit row offsets) until no further
   page token is provided.

[callmehiphop]

This throws me off a little bit, does this imply that the best way to go is to poll the getQueryResults API until we get a single page of results and only then should we call listTableData?

[steffnay]

Yeah, that's what Seth told me they do in the other languages and how I should handle it to be consistent.

[callmehiphop]

We're probably going to need to cache the destinationTable in case users don't want to autoPaginate the results