Support extraction of 'large' (>= ~2 GiB) files #38
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The behaviour of `syswrite` depends on the platform and seems to be different when getting closer to writing about 2 GiB or more at once. On Linux, it will write at most (2**31 - 4096) bytes [1,2] and not return an error when more data was passed in but just return the amount of data that was actually written - so the original implementation was producing incomplete/corrupt files during extraction when they were larger than (2**31 - 4096) bytes. On macOS, the limit appears to be (2**31 - 1) bytes, otherwise, an error is returned. So in order to correctly extract files close to or larger than 2 GiB, it's necessary to write less than about 2 GiB at once and redo write operations until all data actually has been written. [1] https://www.man7.org/linux/man-pages/man2/write.2.html#NOTES [2] https://stackoverflow.com/questions/70368651/why-cant-linux-write-more-than-2147479552-bytes
|
Thanks for merging and releasing this! I did notice though that the tests I added seem to cause some CPAN testers test failures on 32 bit Perls since the largest block sizes ( archive-tar-new/t/02_methods.t Lines 591 to 592 in dcf41a3 2**31 - 1. If you like, I can create a follow-up PR with such a fix.
|
|
That would be awesome, many thanks. I had seen the test failures but lacked tuits to start to dig into what was going on. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When extracting Tar archives with 'large' files (i.e. files larger than ~2 GiB) I noticed that this either didn't work (macOS) or resulted in corrupt/too small files (Linux). While debugging, I noticed that there are limits to the amount of data that
syswritewrites, on Linux, it's at (2**31 - 4096) bytes while on macOS, it appears to be at (2**31 - 1) bytes. In addition to that,syswriteon Linux doesn't return an error in that case, but just the amount of data actually written (which is well within the specified behavior ofwrite(2)), while on macOS, it returns an error - which explains the behavior I observed. There's also an older bug report on rt.cpan.org which seems to describe the same issue on Windows.This PR changes the code which writes extracted files to (1) write smaller chunks (1 GiB) and (2) write until all data was actually written. This seemed to fix the problem in my tests with the file where the extraction failed before. I'm not sure if this is the best solution for the problem, but at least on Linux, the current implementation seems problematic since it may produce incomplete files without warning or error.