failed to add query result to dynamic destination table (on a simple query)

[Cililing]

What happened?

I have a table, pretty big, over 4M rows (if it's important the table contains arrays with simple types, like REPEATED/INTEGER).

So, I am making a query:
bq --api http://0.0.0.0:9050 query --project_id=company "SELECT count(*) FROM team.Items"

(other queries are affected too, for example:

{"query": "SELECT avg(orderedQuantity) FROM team.Items", "values": []}

and getting response:

BigQuery error in query operation: failed to add query result to dynamic destination table: dataset bqjob_r5323c5b3273d9d3f_0000018e1d743054_1 is already created

Logs on the emulator are:

2024-03-08T10:45:37.660+0100	INFO	contentdata/repository.go:167		{"query": "SELECT count(*) FROM team.Items", "values": []}
2024-03-08T10:45:58.684+0100	DEBUG	contentdata/repository.go:246	query result	{"rows": [[4449208]]}
2024-03-08T10:45:58.705+0100	INFO	server/middleware.go:63	POST /projects/company/jobs	{"query": "alt=json"}
2024-03-08T10:45:58.715+0100	INFO	contentdata/repository.go:167		{"query": "SELECT count(*) FROM team.Items", "values": []}
2024-03-08T10:46:13.453+0100	DEBUG	contentdata/repository.go:246	query result	{"rows": [[4449208]]}
2024-03-08T10:46:13.453+0100	ERROR	server/handler.go:1018	jobInternalError	{"error": "jobInternalError: failed to add query result to dynamic destination table: dataset bqjob_r5323c5b3273d9d3f_0000018e1d743054_1 is already created"}

What did you expect to happen?

I'm expecting the result to be returned.

How can we reproduce it (as minimally and precisely as possible)?

• Prepare table with over 4M rows
• Execute query (python API / bq client)
• Error occures every time

Anything else we need to know?

No response

[ohaibbq]

It seems to me like the BigQuery client is retrying your query as the server closed the connection.
Judging by the timestamps in your logs, I'm guessing you're running into this issue: #213 / #266
@goccy Do you have any opinions on #266?

[Cililing]

BQ client allows specifying the job timeout (--job_timeout_ms=100000), which is ignored by the emulator.

The python lib has this possibility as well:

def query(
        ....
        timeout: TimeoutType = DEFAULT_TIMEOUT
    )
---
DEFAULT_RETRY = retry.Retry(predicate=_should_retry, deadline=600.0)
"""The default retry object.

Any method with a ``retry`` parameter will be retried automatically,
with reasonable defaults. To disable retry, pass ``retry=None``.
To modify the default retry behavior, call a ``with_XXX`` method
on ``DEFAULT_RETRY``. For example, to change the deadline to 30 seconds,
pass ``retry=bigquery.DEFAULT_RETRY.with_deadline(30)``.
"""

DEFAULT_TIMEOUT = None
"""The default API timeout.

This is the time to wait per request. To adjust the total wait time, set a
deadline on the retry object.
"""

[ohaibbq]

This should be closed now that #266 has been merged.

[ohaibbq]

Did the new release fix your issue? @Cililing