Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: tries to insert NaN for source_table_name in GroupedQuery count #50

Closed
2 tasks done
tswast opened this issue Jul 15, 2020 · 4 comments · Fixed by #64
Closed
2 tasks done

BUG: tries to insert NaN for source_table_name in GroupedQuery count #50

tswast opened this issue Jul 15, 2020 · 4 comments · Fixed by #64
Assignees

Comments

@tswast
Copy link
Collaborator

tswast commented Jul 15, 2020

When count doesn't exist in a group, some columns end up as NaN. May need to filter these out when inserting rows from dataframe.

TODO:

@tswast tswast self-assigned this Jul 15, 2020
@tswast
Copy link
Collaborator Author

tswast commented Jul 15, 2020

Filed issue with google-cloud-bigquery: googleapis/python-bigquery#169

This is a general problem with uploading dataframes that use NaN as a NULL indicator. In the meantime we could work around by filtering out NaN values before uploading / converting NaN to None, but I think that's best done in the google-cloud-bigquery library.

@tswast
Copy link
Collaborator Author

tswast commented Jul 15, 2020

Fix pending at googleapis/python-bigquery#170

@tswast
Copy link
Collaborator Author

tswast commented Jul 17, 2020

Fix merged. Waiting on release googleapis/python-bigquery#173

@tswast
Copy link
Collaborator Author

tswast commented Jul 21, 2020

Possible workaround while waiting for BigQuery release: https://stackoverflow.com/a/54403705/101923

df = df.replace({np.nan: None})

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant