Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimization of the Staging Area should index the data #294

Open
osopardo1 opened this issue Mar 26, 2024 · 0 comments
Open

Optimization of the Staging Area should index the data #294

osopardo1 opened this issue Mar 26, 2024 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@osopardo1
Copy link
Member

osopardo1 commented Mar 26, 2024

Qbeast Format supports reading files not indexed with the Otree / not containing Qbeast Metadata. We call it the "Staging Area".

To reduce the overhead of writing small files, we use an operation called Compaction on this area. But changes on #284 would reduce the implementation of compaction to a single call to the optimize method. On the other hand, the optimization of the Staging Area should NOT be the same than the optimization for Indexed Files.

The solution proposed is:

  1. Delete compact() as an actionable command both in IndexedTable and QbeastTable.
  2. Remove CompactTableCommand.
  3. Remove or fix tests accordingly.
  4. Rename compact() method in the DataWriter interface and implementation. Might be confusing to still use that naming internally.
  5. When optimize() is applied to the Staging Area -> index the data.
  6. When optimize() is applied to other areas / revisions -> maintain the current implementation.
@osopardo1 osopardo1 added the bug Something isn't working label Mar 26, 2024
@osopardo1 osopardo1 self-assigned this Apr 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant