Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for changing indexed columns on append #172

Open
3 tasks
osopardo1 opened this issue Mar 15, 2023 · 1 comment
Open
3 tasks

Support for changing indexed columns on append #172

osopardo1 opened this issue Mar 15, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@osopardo1
Copy link
Member

The current implementation of the Qbeast Format on Spark does not support appending new data with a different indexed columns that the ones specified at the beginning.

Since data is constantly evolving, and business needs may change, it does makes sense that Qbeast index goes in the same direction. Also, it is designed in a way that it can support different index versions (called Revisions), and querying each version independently.

The to-do's of this issue are:

  • Remove constraints that checks wether the columnsToIndex are equal to the current indexed columns.
  • Check Spark Plan to build the qbeast_hash accordingly. When we remove totally the hash (as proposed in Overhead of qbeast_hash filtering when doing a Sample #68 ), this would no longer be needed.
  • Test and check the correctness of the results when doing sampling, filtering and reading the whole data.
@osopardo1 osopardo1 added bug Something isn't working enhancement New feature or request status: on-hold This issue is on hold and removed bug Something isn't working labels Mar 15, 2023
@osopardo1
Copy link
Member Author

On hold due to changes on #175 and #68 . Closing both issues would affect directly to this implementation.

@osopardo1 osopardo1 removed the status: on-hold This issue is on hold label Oct 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant