pack: Add support for using a common pack dir for multiple traces #3735
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When packaging traces from CI, it's fairly commong to have hundreds of traces that all basically share the exact same files. This can lead to some fairly large traces after packing. Of course, some file-systems support block-level deduplication and a compression library would certainly be able to dedup it back down as well, but it'd be faster to not create trace directories that big on disk in the first place. This adds a `--pack-dir` command to `rr pack <traces...>`, which is used as a the common pack dir for all traces. Rather than packing files into their own trace dirs, they will be packed into the `pack-dir`, with relative symlinks from the original trace directories to the pack dir. An unmodified rr will be able to replay these as long as the pack dir is moved along with the trace dirs.
Keno
added a commit
to JuliaPackaging/Yggdrasil
that referenced
this pull request
Apr 24, 2024
This bumps rr to current master, with rebased xcr0 patches. It drops the `patch --index-dir` patch which was rejected upstream and we do not use (as far as I know). The `pack --pack-dir` patch is rebased and resubmitted upstream as rr-debugger/rr#3735.
maleadt
pushed a commit
to JuliaPackaging/Yggdrasil
that referenced
this pull request
Apr 25, 2024
This bumps rr to current master, with rebased xcr0 patches. It drops the `patch --index-dir` patch which was rejected upstream and we do not use (as far as I know). The `pack --pack-dir` patch is rebased and resubmitted upstream as rr-debugger/rr#3735.
rocallahan
reviewed
Apr 28, 2024
| @@ -13,6 +13,7 @@ | |||
| #include <unistd.h> | |||
|
|
|||
| #include <algorithm> | |||
| #include <filesystem> | |||
There was a problem hiding this comment.
I think this will require gcc 9.1 and stop rr from building on Ubuntu 18 systems, which I believe works currently. Can we just reimplement filesystem::relative, or is that hard?
| @@ -528,7 +536,8 @@ static map<string, string> compute_canonical_symlink_map( | |||
| * for all files with that hash. | |||
| */ | |||
| static map<string, string> compute_canonical_mmapped_files( | |||
| const string& trace_dir) { | |||
| const string& trace_dir, | |||
| PackDir &pack_dir) { | |||
There was a problem hiding this comment.
& before the space please
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When packaging traces from CI, it's fairly commong to have hundreds of traces that all basically share the exact same files. This can lead to some fairly large traces after packing. Of course, some file-systems support block-level deduplication and a compression library would certainly be able to dedup it back down as well, but it'd be faster to not create trace directories that big on disk in the first place.
This adds a
--pack-dircommand torr pack <traces...>, which is used as a the common pack dir for all traces. Rather than packing files into their own trace dirs, they will be packed into thepack-dir, with relative symlinks from the original trace directories to the pack dir. An unmodified rr will be able to replay these as long as the pack dir is moved along with the trace dirs.We've had this feature on the JuliaLang fork for a while, but it was tangled up with #2529 and I don't believe I ever PR'd it separately. I think there's a stronger case for this part of the feature, because it doesn't require any unpacking steps on the receiving side and is best done during the packing process.