DBAPI does not add tracing spans to additional page fetches

[nickmarx12345678]

Environment details

• OS type and version: Darwin M-C02D404FMD6T 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:54:10 PST 2023; root:xnu-10002.61.3~2/RELEASE_X86_64 x86_64
• Python version: Python 3.11.3
• pip version: `pip 22.3.1
• google-cloud-bigquery version: Version: 3.15.0

Steps to reproduce

I will try to add a self contained repro, but I'm newer to this library/python, so I will summarize in the meantime.

We have been using the opentelemetry integration along with GCP Cloud Trace to reduce latency of an application using python-bigquery and have noticed when there is more than one page worth of results in a DB-API cursor, we will only see trace spans for the initial bq.query and job.reload calls, but not for the fetching of the additional pages, which leaves an unexplained gap in the trace timeline (see pic)

To reproduce, run a DB-API query with array_size

python-bigquery/google/cloud/bigquery/dbapi/cursor.py

Line 76 in 955a4cd

self.arraysize = None

set to less than the output number of rows of the query, and opentelemetry tracing installed, and observe .

Code example

cursor = execute(sql)
results = cursor.all()

subset of flame graph from my debugging. this segment of calls appears not to generate any tracing spans (despite using bigquery_storage_client)

Thanks!

[tswast]

There are two methods in which we download results: REST API (this client library) and BQ Storage Read API.

For REST API, it appears we need to add the span attributes in the following location:

python-bigquery/google/cloud/bigquery/client.py

Line 3943 in 955a4cd

api_request=functools.partial(self._call_api, retry, timeout=timeout),

For the BQ Storage API, please file a feature request for opentelemetry support in our gRPC clients here: https://github.com/googleapis/gapic-generator-python/issues?q=is%3Aissue+is%3Aopen+

[nickmarx12345678]

Done! googleapis/gapic-generator-python#1910

Thanks @tswast