Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-threading / multi-threaded support for multi-core system #491

Closed
Hedda opened this issue Dec 19, 2016 · 4 comments
Closed

Multi-threading / multi-threaded support for multi-core system #491

Hedda opened this issue Dec 19, 2016 · 4 comments
Assignees

Comments

@Hedda
Copy link

Hedda commented Dec 19, 2016

Please consider adding support for multithreading / multithreaded compresion and decompression for faster processing on systems with multiple CPU cores.

@terrelln
Copy link
Contributor

pzstd is a command line tool that does multithreaded compression and decompression. The pzstd format uses zstd skippable frames to encode the information it needs for parallel decompression, so pzstd compressed data is compatible with standard zstd decompression.

Adding parallel compression to the zstd binary is planned. Issue #471 also tracks the addition of a parallel library function.

@ghost
Copy link

ghost commented Dec 20, 2016

There is also an zstdmt library. But it's API is not considered stable currently... also the documentation is not finished.

@FrancescAlted
Copy link

FrancescAlted commented Dec 20, 2016

FWIW, the Blosc library also has a multithreaded implementation for a series of codecs (currently BloscLZ, LZ4, LZ4HC, Snappy and Zstandard), plus support for shuffling filters with support for SSE2 and AVX2 SIMD instruction set in Intel-compatible processors.

@Cyan4973 Cyan4973 self-assigned this Feb 6, 2017
@Cyan4973
Copy link
Contributor

Cyan4973 commented Feb 6, 2017

Multi-threading support has been added to latest release v1.1.3.

Since it's still experimental, it needs to be explicitly targeted (make zstdmt from root file).
Then it's possible to request nb of threads from the command line, for example :
zstd -T4 file

@Cyan4973 Cyan4973 closed this as completed Feb 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants