BigQuery storage client error when downloading to pandas dataframe

[YvesJacquot]

Hi,

I am trying to download query results using the BigQuery client library, as per instructions on this page.

import google.auth
from google.cloud import bigquery
from google.cloud import bigquery_storage_v1beta1
# Make clients.
bqclient = bigquery.Client(
    credentials=credentials,
    project=your_project_id,
)
bqstorageclient = bigquery_storage_v1beta1.BigQueryStorageClient(
    credentials=credentials
)

# Download query results.
table_name = 'project.dataset.table'
query = f'''\
SELECT *
FROM   `{table_name}`
'''
dataframe = (
    bqclient.query(query)
    .result()
    .to_dataframe(bqstorage_client=bqstorageclient)
)
print(dataframe.shape)

I get the following error message.

File "/Users/**/anaconda3/lib/python3.7/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 618, in _download_table_bqstorage
read_options = bigquery_storage_v1beta1.types.TableReadOptions()
AttributeError: 'NoneType' object has no attribute 'types'

Any idea how I could fix this?

I have the following config:

macOS High Sierra Version 10.13.6
Python 3.7.6.
google-cloud 0.34.0
google-cloud-bigquery 1.22.0
google-cloud-bigquery-storage 0.7.0

Many thanks

[IlyaFaer]

Looking at this lines, I'd say you just need to install BigQuery Storage library:

google-cloud-python/bigquery/google/cloud/bigquery/_pandas_helpers.py

Lines 24 to 27 in 0384238

	try:
	from google.cloud import bigquery_storage_v1beta1
	except ImportError: # pragma: NO COVER
	bigquery_storage_v1beta1 = None

See this folder:
https://github.com/googleapis/google-cloud-python/tree/master/bigquery_storage/google/cloud

[IlyaFaer]

@tswast, I believe, such a failure should be covered with some kind of a message, 'cause no attribute seems not very descriptive to me. WDYT?

[YvesJacquot]

Hi
Yes it seems like _pandas_helpers.py doesn't manage to import the BigQuery Storage library.
However I did install it (version 0.7.0.).
What should I look for in the folder you mentioned?
Thank you

[IlyaFaer]

@YvesJacquot, well, this is just the directory you should install
Did you installed it like described in client readme, with virtualenv?
https://github.com/googleapis/google-cloud-python/blob/master/bigquery_storage/README.rst
I think, this should be enough to make things work. If it still fails, we should take a look at import error traceback (maybe there is something more missing).

[YvesJacquot]

@IlyaFaer, yes, installing it in a virtual environment solved it
Thank you for your help!

[piktiv-ilkin]

I was using this example to get data from my dataset and got same error. I deleted bqstorage_client=bqstorageclient param from to_dataframe function and it solved my problem. New code:

dataframe = (
    bqclient.query(query_string)
    .result()
    .to_dataframe()
)

I'm using AI-Platform Notebooks server, with

• python 3.7.6
• google-cloud-bigquery==1.25.0
• google-cloud-bigquery-storage==1.0.0

[kstamatiou]

I confirm that I had to delete bqstorage_client=bqstorageclient from this example in order to get it to work.

[ivanmkc]

@tswast I can also confirm that I had to delete bqstorage_client=bqstorageclient for it to work. Seems like a bug.

I didn't require this fix when running my script locally, but when running on Vertex AI training, it didn't work until I used this fix. Ping me for the code.

[mohanwd]

I have also faced the same issue. below changes have helped me.

After we updated our google libraries, there is no need to specify the BigQuery Storage API explicitly.

changed line below to next one and it worked.

df= (query_job.result().to_dataframe(bqstorage_client=bqstorageclient))
#above line is used in deprecated version

df= (query_job.result().to_dataframe(create_bqstorage_client=True))
#This one works

Ref : https://cloud.google.com/bigquery/docs/bigquery-storage-python-pandas#download_table_data_using_the_client_library