Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make build of cpio reproduceable #323

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Conversation

mslacken
Copy link
Member

No description provided.

@@ -16,7 +16,7 @@ rootfs:
done

capability.cpio: rootfs
cd rootfs/; find . | cpio -o -H newc -F ../capability.cpio
cd rootfs/; find . | xargs touch -d @1690848000 ; find . | sort | cpio -o -H newc -F ../capability.cpio
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you changing the timestamp on everything to Aug 1, 2023 12:00am ?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jmstover Presumably so two given builds will both have the same timestamps and thus result in the same byte for byte cpio... assuming of course that the contents are the same as well.

It's too bad cpio or bsdcpio itself doesn't have a way todo this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I understand that. But we aren't going for byte for byte identical here... I mean, I've created different capabilities for a single set of servers than other servers on the same cluster... It's the functionality a capability does that's the important thing (IMO). So, I'm just trying to figure out why ... ;)

Copy link
Contributor

@jmstover jmstover Dec 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To expand... I don't see an overall issue with it. I don't think changing the date across the board should break anything in the bootstraps ... I'd just like to know why a given change is being done for my peace of mind. =)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for reproducible builds as @bsallen-argonne guessed it right. Distributions like openSUSE like the idea, that the same sources lead to the same binaries.
I should have mentioned this in the PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

anyone know of a better way to set all 3 timestamps and/or get cpio to set it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think with touch you can (unless you run it without the -d option, then it changes all 3). Running touch -t [...] only changes Access and Modify times. I know cpio can reset the access time, so you don't know the file has been read... but I don't think there's a way to set the timestamp to something specific.

As ugly as it would be, the only way I can see us doing this would be to take Greg's cpio code from wwvnfs, and modify it to see if we can set atime and ctime ... I know it sets mtime, but don't think anything else .... :/

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From looking around a bit more, it looks like ctime is supposed to be extremely difficult to arbitrarily change. One way to do it is to use debugfs on a unmounted filesystem... But, beyond that POSIX says atime/mtime are user-setable ... but ctime the system must tack unerringly.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mslacken Ohhhhh.... I just noticed this was on master branch. Can you rebase it to development?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bit this pr goes to the master branch, so the PR should also go to the development branch instead?

jmstover
jmstover previously approved these changes Dec 20, 2023
Copy link
Contributor

@jmstover jmstover left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Attempt at reproducible build seems fine. I don't think the change should interfere with the bootstrap at all.

bensallen
bensallen previously approved these changes Dec 20, 2023
@bensallen bensallen changed the base branch from master to development January 2, 2024 15:45
@bensallen bensallen dismissed stale reviews from jmstover and themself January 2, 2024 15:45

The base branch was changed.

@bensallen bensallen changed the base branch from development to master January 2, 2024 15:45
Signed-off-by: Christian Goll <cgoll@suse.com>
@e4t
Copy link

e4t commented Mar 25, 2024

Unfortunately, there are more places which require adjustments: there are 8 sub-directories which contribute to this cpio file and there is at least one other cpio file. All of these need to be included. I'm investigating.

@e4t
Copy link

e4t commented Mar 27, 2024

Unfortunately, there are more places which require adjustments: there are 8 sub-directories which contribute to this cpio file and there is at least one other cpio file. All of these need to be included. I'm investigating.

I've looked into this: Using touch to set the access and modification time of a files to a fixed value is sufficient to make cpio archives reproducible - if cpio is called with --reproducible. We have verified this here.
However, the changes required to make this work across the all of warewulf3 are more extensive than what's been submitted.
I will create a separate pull request with the full set of changes.

@e4t e4t mentioned this pull request Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants