Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Empty type decoding: move the segment buffer forward by the compressed data size for empty types #1462

Draft
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

muhammadhamzasajjad
Copy link
Collaborator

@muhammadhamzasajjad muhammadhamzasajjad commented Mar 26, 2024

Reference Issues/PRs

There is a compatibility issue with columns containing empty types written in 4.0.x or earlier.
For version 4.0.x, the buffer was remaining behind the correct position which was causing an lz4 decoding error.

What does this implement or fix?

Any other comments?

Checklist

Checklist for code changes...
  • Have you updated the relevant docstrings, documentation and copyright notice?
  • Is this contribution tested against all ArcticDB's features?
  • Do all exceptions introduced raise appropriate error messages?
  • Are API changes highlighted in the PR description?
  • Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?

@poodlewars
Copy link
Collaborator

We should add a test that reproduces the original problem (ie writes with 4.0.x and updates with HEAD).

@G-D-Petrov
Copy link
Collaborator

We should add a test that reproduces the original problem (ie writes with 4.0.x and updates with HEAD).

Test should look something like this:

# version 4.0.3
df = pd.DataFrame({'c1': [None, None, None], 'c2': [1.2, None, 1.3]}, index=pd.date_range('1/1/2018', end='1/03/2018'))
lib.write("mysym", df)

# version latest master
df = pd.DataFrame({'c1': [None, None, None], 'c2': [1.1, None, 1.4]}, index=pd.date_range('1/1/2018', end='1/03/2018'))
lib.update("mysym", df)
...

arcticdb_ext.exceptions.InternalException: E_INVALID_ARGUMENT Error while decoding with lz4 at address 7eacd8d4efea with size 21. Code -4 // this error shows  failure in decoding something written with 4.0.3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants