Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ToplingZipTable Builder: support distributed compressing #47

Open
rockeet opened this issue May 24, 2023 · 0 comments
Open

ToplingZipTable Builder: support distributed compressing #47

rockeet opened this issue May 24, 2023 · 0 comments

Comments

@rockeet
Copy link
Collaborator

rockeet commented May 24, 2023

Compaction needs CompactionFilter, which may use DB::Get for metadata(such as pika/todis/kvrocks), in distributed compaction, compact_worker has no DB object, thus can not support such compaction.

ToplingZipTable Builder use two-pass scanning, it save decompressed kv data into tmp files, in second pass scaning, it read data from tmp file, thus we can run first pass scaning in DB side(local compation), and run second pass scanning in compaction worker to compress data -- compressing consumes 80+% CPU time for ToplingZipTable.

@rockeet rockeet changed the title ToplingZipTable Builder: support remote compression ToplingZipTable Builder: support distributed compression May 24, 2023
@rockeet rockeet changed the title ToplingZipTable Builder: support distributed compression ToplingZipTable Builder: support distributed compressing Nov 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant