Improve performance for multi-threaded access to encrypted zip files

[mxmlnkn]

#96 (comment)

BTW one thing i've discovered when trying to integrate libarchive is that python zipfile has similar inefficiencies as tarfile module:
if 2 threads trying to access the same member it will decompress (and decrypt if pw-protected) the member from the beginning.
The situation is better than with .tar.gz when it needs to decompress from the start of the whole archive, but still poblematic.
There is a need to develop something like SQLindexedTar class to checkpoint decompression and decryption states..

This might need yet another backend like indexed_bzip2 that works with zip files. So... a lot of work.

[mxmlnkn]

@Vadiml1024 Could it be that you are running into this issue instead:

Decryption is extremely slow as it is implemented in native Python rather than C.

With #98 also observing performance issues I get the feeling that a better zip module must be available :/. Maybe libarchive? But, we tried. And, concurrency support in the libarchive Python-bindings was a work in progress. czipfile exists but it seems to be Python 2 and dead.

So, I guess another self-written backend.

[Vadiml1024]

I'look I to czipfile I suppose porting it to python3 will be not too difficult.. Le mar. 22 nov. 2022, 21:15, Maximilian Knespel ***@***.***> a écrit :

…

@Vadiml1024 <https://github.com/Vadiml1024> Could it be that you are running into this issue <https://docs.python.org/3/library/zipfile.html> instead: Decryption is extremely slow as it is implemented in native Python rather than C. With #98 <#98> also observing performance issues I get the feeling that a better zip module must be available :/. Maybe libarchive? But, we tried. And, concurrency support in the libarchive Python-bindings was a work in progress. czipfile <https://pypi.org/project/czipfile/> exists but it seems to be Python 2 and dead. So, I guess another self-written backend. — Reply to this email directly, view it on GitHub <#97 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG76GLFVDCK2NRECIXVY5DWJUSUNANCNFSM6AAAAAAR7DDISI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Vadiml1024]

Somebody already ported czipfile to python3
https://github.com/ziyuang/czipfile

[mxmlnkn]

Ah nice. I didn't see it on PyPI.

Cython as opposed to Python is also said to be faster: https://stackoverflow.com/a/72513075/2191065

And there is this: https://github.com/TkTech/fasterzip But it seems like it might be missing some features like setting a password among others.