Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OneOf fields skipped as "synthetic" but still marshalled by the Marshaller #368

Open
ubunatic opened this issue Jan 11, 2024 · 1 comment · May be fixed by #369
Open

OneOf fields skipped as "synthetic" but still marshalled by the Marshaller #368

ubunatic opened this issue Jan 11, 2024 · 1 comment · May be fixed by #369

Comments

@ubunatic
Copy link

We have upgraded from v0.25.0 to a higher version lately and see issues with newly created tables.
We use options.InferSchema(msg) to create and update schemas for our tables.
We use options.Marshal(msg) to marshal the messages.

With version v.26.0+, some rows cannot be ingested and BQ rejects them with an error:

some.field._some_one_of: no such field: _some_one_of."; Reason: "invalid"

We use these Options:

protobq.SchemaOptions{
	UseEnumNumbers:           false,
	UseOneofFields:           true,
	UseDateTimeWithoutOffset: true,
}

When calling options.Marshal(msg), the nested map[string]bigquery.Value will contain the _some_one_of field.
When calling options.InferSchema(msg) the nested bigquery.Schema will not contain the _some_one_of field anymore.

When I disable this change:

  // if oneof.IsSynthetic() {
  // 	continue
  // }

everything works as expected.

How do you handle this case?

@ubunatic ubunatic linked a pull request Jan 12, 2024 that will close this issue
@ubunatic
Copy link
Author

I added a PR: #369 to fix it. Let me know if this works for you.
Starting next week, we will vendor our branch with the fix to fill our DWH. 🤞🏼
I will post the results here too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant