issue161/BigQuery BIGNUMERIC column error #162
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tswast
reviewed
Dec 10, 2020
tswast
reviewed
Dec 22, 2020
| # TODO(dhercher): DataType customizations shoould be moved to Ibis/third_party | ||
| # BigQuery BIGNUMERIC support needs to be pushed to Ibis | ||
| bigquery._pandas_helpers.BQ_TO_ARROW_SCALARS["BIGNUMERIC"] = pyarrow.string | ||
| _DTYPE_TO_IBIS_TYPE["BIGNUMERIC"] = dt.string |
There was a problem hiding this comment.
Could you describe why BIGNUMERIC and NUMERIC are handled differently?
Also, I suspect you want float64 because Ibis has more aggregate functions implemented there than they do for the decimal data types? Could you comment on why we need to remap it here?
There was a problem hiding this comment.
An error occurs with float when it is attempted to be read to pandas. I believe this needs to wait until pyarrow. and full support is added too
Co-authored-by: Tim Swast <swast@google.com>
…gleCloudPlatform/professional-services-data-validator into issue161-support-bigquery-bignumeric
renzokuken
approved these changes
Jan 6, 2021
| @classmethod | ||
| def to_ibis_from_I2(cls, col_data, return_ibis_type=True): | ||
| if return_ibis_type: | ||
| return dt.int8 |
There was a problem hiding this comment.
what's the difference between to_ibis_from_I1 and to_ibis_from_I2
There was a problem hiding this comment.
I1 and I2 are the teradata datatypes for different int values, both are encapsulated by int8 though
freedomofnet
added
Beta
priority: p0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The BIGNUMERIC datatype causes an error to be thrown in Ibis when getting schema. We can easily resolve this with the code supplied here to add Ibis support for BIGNUMERIC.
However, ideally the field should be defined as a numeric type. This would allow numeric aggregations and functions o be applied in Ibis. When we attempt to use
dt.Numeric(38,9)then we get an error from the BigQuery client when executing the query.We are suggesting a temporary
dt.stringtype to unblock schema conversion. However, this does not resolve the issue which will still crop up