Build reproducibility problem: "git archive" locally results in different .tar.gz than on GitHub #1279
Labels
Comments
rugk
changed the title
rugk
changed the title
rugk
changed the title
|
Hey, thanks for testing it. I'll also raise a PR over on your repo with my local script mods I had to do over the years, shortly. In the mean time, 9a476ac is not tag 1.7.1, which is aad975a instead. If I checkout tag 1.7.1 (detached HEAD-less state) and run git archive I do get the same archive as github: Please verify on your end, if it is really just the difference in checked out git commit. I would greatly, immensely appreciate it if you could verify that you are all setup and able to handle this process, too, should we need to urgently release something and I'm on holiday or otherwise absent. |
|
A thanks wrong repo version, of course! (I thought my script checked out the tag, I was wrong of course.) Uhm, I do get different git hashes here, but I wonder why… $ git checkout 1.7.1
HEAD ist jetzt bei aad975a7 incrementing version
$ git switch --detach 1.7.1
HEAD ist jetzt bei aad975a7 incrementing version
$ /signrelease.sh
Executed in /var/home/rugk/Software/PrivateBin/PrivateBin.
NOTE: You already need to have a published release on GitHub.
Enter the project name [PrivateBin]:
Enter the tag to sign: 1.7.1
Paste GitHub URL here [https://github.com/PrivateBin/PrivateBin]:
gpg: Signatur vom So 11 Feb 2024 15:32:00 CET
gpg: mittels RSA-Schlüssel 1C2A890AF1135CEC3681666A0F5C940A6BD81F92
gpg: Korrekte Signatur von "El RIDO (key for signing the git commits of the PrivateBin project) <elrido@gmx.net>" [unbekannt]
gpg: WARNUNG: Dieser Schlüssel trägt keine vertrauenswürdige Signatur!
gpg: Es gibt keinen Hinweis, daß die Signatur wirklich dem vorgeblichen Besitzer gehört.
Haupt-Fingerabdruck = 1C2A 890A F113 5CEC 3681 666A 0F5C 940A 6BD8 1F92
git archive --prefix=PrivateBin-1.7.1/ -o /tmp/signrelease/PrivateBin-1.7.1-a5oKhJxSqs/PrivateBin-1.7.1.tar.gz 1.7.1
git archive --prefix=PrivateBin-1.7.1/ -o /tmp/signrelease/PrivateBin-1.7.1-a5oKhJxSqs/PrivateBin-1.7.1.zip 1.7.1
Binärdateien /tmp/signrelease/PrivateBin-1.7.1-a5oKhJxSqs/PrivateBin-1.7.1.tar.gz und /tmp/signrelease/PrivateBin-1.7.1-a5oKhJxSqs/GitHubDownloadedArchive.tar.gz sind verschieden.
FATAL ERROR: GitHubs downloaded tar.gz archive file is different from our own.
$ sha256sum *.tar.gz
136faa5ba0c2d51ace916a2c0b282f5c7d5376265406040c91730e28233367ff GitHubDownloadedArchive.tar.gz
ca7274e87ee621ca524e9457a4e6347f34286ac4680968431812cca3e870859d PrivateBin-1.7.1.tar.gzSo, huh, I still get Also, if it were a difference in the files itself/the content, the diffscope should have told me. I really guess it must be something else… |
|
This was on an ubuntu 22.04 system: It sounds very odd that you'd get the exact same Let me repro this again, in a fresh alpine and ubuntu containers with fresh git clones, so we can rule out further environmental factors. |
|
Tried searching for changelog entries etc. But it appears, such problems are well known… The thing is, it just looks as if my local version has changed, not the GitHub one… 😉 Note it could also be my distro patching it… The only thing striking is:
Or wait, did GitHub now opt to keep the old one due to the problems? But we both use newer ones? So we should be either both be able to reproduce the different hash or not…? |
|
The plot thickens: First, turns out that you don't actually need to have anything checked out, the last argument to git archive tells it which commit, branch or tag to export. Second, notice that your git (2.44) keeps generating the Third, here is what I get on alpine: So here I suddenly get that same That really starts to sound like version related differences. Further evidence: |
|
Reading the github blog closer, that you linked: They did change the git archive generation, but...
So I simply didn't notice because my ubuntu git was always below 2.38 and I didn't happen to do a release during the above time frame that github returned different archives. Our container image builds during that time might have failed, though. On to workarounds for newer git versions. We can configure it to still use an external compression command as follows (again tested in alpine, with git 2.43): |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Due to some minor changes in my
signrelease.shscript I tried testing that out again, locally.Steps to reproduce
git statusreports nothing)Now executing
signrelease.sh(in my case v rugk/gittools@35b6f16):What happens
The .tar.gz is different from GitHub grr…
Here are the files:
GitHubDownloadedArchive.tar.gz
PrivateBin-1.7.1.tar.gz
I uploaded them to https://try.diffoscope.org/qyqtqvygkghu.html (stable/archive link) even, and it says:
…which is not really helpful.
Aka:
$ file *.tar.gz GitHubDownloadedArchive.tar.gz: gzip compressed data, from Unix, original size modulo 2^32 2631680 PrivateBin-1.7.1.tar.gz: gzip compressed data, from Unix, original size modulo 2^32 2631680What should happen
Files should be same.
Maybe my git version matters or how it is compressed?
More information
If I modify the script to nevertheless generate the ZIP and compare it, they are identical (different test run obviously here):
I also checked the timezone, as this had been an issue before, but in this case (as for
.tar.gzfiles), this does not seem to make a difference:$ sha256sum *.tar.gz 136faa5ba0c2d51ace916a2c0b282f5c7d5376265406040c91730e28233367ff GitHubDownloadedArchive.tar.gz ca7274e87ee621ca524e9457a4e6347f34286ac4680968431812cca3e870859d PrivateBin-1.7.1-GMT-TZ.tar.gz ca7274e87ee621ca524e9457a4e6347f34286ac4680968431812cca3e870859d PrivateBin-1.7.1.tar.gzAlso run the same with rugk/gittools@71a2d6a, the result is the same (hash also the same again).
System
$ rpm-ostree status -v […] fedora:fedora/39/x86_64/silverblue (index: 1) Version: 39.20240325.0 (2024-03-25T00:37:19Z) […] $ uname -a Linux **** 6.7.9-200.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 6 19:35:04 UTC 2024 x86_64 GNU/Linux $ git --version git version 2.44.0 $ rpm -q git git-2.44.0-1.fc39.x86_64 $ tar --version tar (GNU tar) 1.35 $ rpm -q tar tar-1.35-2.fc39.x86_64 $ gzip --version gzip 1.12 […] $ rpm -q gzip gzip-1.12-6.fc39.x86_64The text was updated successfully, but these errors were encountered: