Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Job created from library doesn't respect project default default_query_job_timeout_ms config #1653

Open
jeremyyeo opened this issue Aug 25, 2023 · 3 comments
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@jeremyyeo
Copy link

jeremyyeo commented Aug 25, 2023

Expected behaviour

BQ jobs created/initated via this library should respect the default_query_job_timeout_ms set on the BQ project.

Issue

Hey team, just raising an inquiry about the second half of my comment raised here dbt-labs/dbt-bigquery#231. From what I can tell, a query created from this library does not respect the default_query_job_timeout_ms set on the BQ project for some reason.

Environment details

  • OS type and version: macOS / Ventura 13.4.1
  • Python version: Python 3.9.13
  • pip version: pip 23.2.1
  • google-cloud-bigquery version:
name: google-cloud-bigquery
Version: 3.11.4
Summary: Google BigQuery API client library
Home-page: https://github.com/googleapis/python-bigquery
Author: Google LLC
Author-email: googleapis-packages@google.com
License: Apache 2.0
Location: /Users/jeremy/src/dbt-basic/venv_dbt_1.5.latest/lib/python3.9/site-packages
Requires: google-api-core, google-cloud-core, google-resumable-media, grpcio, packaging, proto-plus, protobuf, python-dateutil, requests
Required-by: dbt-bigquery

Steps to reproduce

  1. Set the default_query_job_timeout_ms to be 5 minutes:
alter project `cse-sandbox-319708` set options (`region-us.default_query_job_timeout_ms` = 300000);
  1. Run a query/job from the BQ UI that takes > 5 minutes:

image

Notice that we run into an error at 5 minutes - as expected.

  1. Run a query/job from python using this library that takes > 5 minutes (see Python code example below):

image

Notice that on the BQ side - the query/job completes taking > 5 minutes (6 m 41 s) - this is unexpected.

Code example

from google.cloud import bigquery

client = bigquery.Client()

# Perform a query.
query = """
call dbt_jyeo.fake_sleep(400);
select * from `region-us.INFORMATION_SCHEMA.PROJECT_OPTIONS`;
"""

query_job = client.query(query)  # API request
rows = query_job.result()  # Waits for query to finish

for row in rows:
    print(row)

Stack trace

There are no errors per se so nothing to call out here.

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery API. label Aug 25, 2023
@chalmerlowe
Copy link
Contributor

Thanks for pointing this out.

@chalmerlowe chalmerlowe added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. priority: p3 Desirable enhancement or fix. May not be included in next release. labels Aug 28, 2023
@wazi55
Copy link

wazi55 commented Sep 14, 2023

Thanks! Looking into it.

@dataders
Copy link

this is related to #1421, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery API. priority: p3 Desirable enhancement or fix. May not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

No branches or pull requests

4 participants