Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid out of memory issues with parallel xz decoder in some circumstances #81

Open
mxmlnkn opened this issue Mar 19, 2022 · 0 comments
Open
Labels
performance Something is slower than it could be

Comments

@mxmlnkn
Copy link
Owner

mxmlnkn commented Mar 19, 2022

The parallel xz decoder is fast but it doesn't bother thinking much about the used memory. It simply uses a block cache storing in the order of number-of-cores blocks of uncompressed data. For very high compression ratios or large block sizes or even anomalous high compression ratios as might be used by attackers, this might lead to out of memory errors. Ratarmounts priorities are currently on performance not security, so beware when e.g. exposing it as a service and run it in a memory-bound VM or something like that! However, if it not only matters for security but even for legitimate usecases (such as limited memory combined with large but "normal" compressino ratios combined with block sizes chosen too large, e.g., by xz -T), then it should be checked against.

One check for this has already been implemented, namely, the parallel xz decoder is not used, even when requested, if the xz file only has a single block. Because it would not speed it up anyway and only require more memory.

Additionally, the block cache could stop decoding after a cut-off size and then simply decode the block on-the-fly when it is requested instead of caching the uncompressed data.

@mxmlnkn mxmlnkn added the performance Something is slower than it could be label Mar 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Something is slower than it could be
Projects
None yet
Development

No branches or pull requests

1 participant