You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Please consider adding support for multithreading / multithreaded compresion and decompression for faster processing on systems with multiple CPU cores.
The text was updated successfully, but these errors were encountered:
pzstd is a command line tool that does multithreaded compression and decompression. The pzstd format uses zstd skippable frames to encode the information it needs for parallel decompression, so pzstd compressed data is compatible with standard zstd decompression.
Adding parallel compression to the zstd binary is planned. Issue #471 also tracks the addition of a parallel library function.
FWIW, the Blosc library also has a multithreaded implementation for a series of codecs (currently BloscLZ, LZ4, LZ4HC, Snappy and Zstandard), plus support for shuffling filters with support for SSE2 and AVX2 SIMD instruction set in Intel-compatible processors.
Since it's still experimental, it needs to be explicitly targeted (make zstdmt from root file).
Then it's possible to request nb of threads from the command line, for example : zstd -T4 file
Please consider adding support for multithreading / multithreaded compresion and decompression for faster processing on systems with multiple CPU cores.
The text was updated successfully, but these errors were encountered: