Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

consider a linearly scannable archive alternative to kzip #5555

Open
jaysachs opened this issue Mar 28, 2023 · 0 comments
Open

consider a linearly scannable archive alternative to kzip #5555

jaysachs opened this issue Mar 28, 2023 · 0 comments

Comments

@jaysachs
Copy link
Contributor

jaysachs commented Mar 28, 2023

Ingested data is already large, and only getting larger. Being able to process (an alternative to) kzips via a linear scan would significantly reduce I/O costs (both storage and throughput)

Requirements:

  • No new compression or archive scheme; "expandable" / readable via standard archive tools
  • The compilation unit data must precede all other data (which currently only includes file contents)
  • The format should be reasonably compressed

Undecided at present if we'll re-purpose the kzip tool for this, or if we'll create a new tool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant