Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change serialization for ChildFieldIDs #11996

Closed
wants to merge 4 commits into from

Conversation

carlopi
Copy link
Contributor

@carlopi carlopi commented May 10, 2024

Only relevant change: moving from [de]serializing unique_ptr<case_insensitive_map_t<FieldID>> to [de]serializing case_insensitive_map_t<FieldID>, given in this particular case the unique_ptr is always valid it should be OK iff:

  • in this particular case we do not care with compatibility across version for this specific statement (??). This is attenuated by the fact that currently deserialization in this case is broken, so the fact it's not working probably points to the fact that is not relevant. If cross version [de]serialization happens for this statement this can still generate problems when say an old version produces something broken that is then reinterpreted as something different by a new version. Unsure what's the care needed around this.

There are other possible solution to this problem, but giving they require to add surface I went for the minimal fix.
Probably figuring out the underlying cause of failures in serialization, that are somewhere at the interstection of unique_ptr and serialization of vectors, but have not been able to figure out a proper fix there.

I also added two minor improvements (I think): forcing 0 initialization of buffers (although not needed) and enforcing invariant that while pre-fetching bytes to check for field ID no bytes are ever read from the stream. Neither were actual problems, but I think it make sense to add those lightweight changes.

Closes #11985

Copy link
Collaborator

@Mytherin Mytherin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Looks good - one comment:

extension/parquet/include/parquet.json Outdated Show resolved Hide resolved
@Mytherin
Copy link
Collaborator

Superseded by #12030

@Mytherin Mytherin closed this May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants