cross-systems usage with colmena & --build-on-host

[phaer]

Hello,

Thanks for this interesting project & and your general work in the Nix ecosystem!

I am deploying from aarch64-darwin to x86_64-linux with colmena and forceRekeyOnSystem = "aarch64-darwin", and ran into the problem that --build-on-target stops working as colmena wouldn't copy rekeyed secret.

Curios if others here found better workarounds than me?
It works if I either use a remote-builder and leave out ´--build-on-targetor if I copy the secrets manually vianix copy. While doing so, I found that it's hard to know which derivation belongs to which host in the output of nix run .#rekey -- --show-out-paths`. Would a PR for that be welcome?

[oddlama]

[...] and ran into the problem that --build-on-target stops working as colmena wouldn't copy rekeyed secret.

Oh good point, I haven't actually run into this myself since I usually build on my local machine.

Curios if others here found better workarounds than me?

Copying all secret derivations to your own substituter would also work, but that may or may not be desirable depending on your infrastructure setup.

While doing so, I found that it's hard to know which derivation belongs to which host in the output of nix run .#rekey -- --show-out-paths`. Would a PR for that be welcome?

Certainly, contributions are always welcome! :D

I can think of either extending the .#rekey CLI to be able to output hostnames, or maybe even better by exposing the rekey derivation in the module directly as a readOnly option age.rekey.drv or something similar to that. How would your desired interface look like?

UPDATE: The resulting derivation can now be targeted via age.rekey.derivation.

[NyCodeGHG]

I think I'm running into a similar issue when deploying from x86_64-linux to aarch64-linux with building on the remote.
Is there a workaround for this? I'm not really sure on how to fix that problem

[oddlama]

@NyCodeGHG Your issue is probably the following: Your local system must build the derivation containing the rekeyed secrets for your target host, since it's the only system that has access to your secret key. But initiating the true build process on the remote means that the remote doesn't know about your local system at all, so naturally it cannot fetch the rekeyed secrets in any way.

There are several workarounds/solutions to this:

1. Don't build on the remote directly, but configure your aarch64-linux as a remote builder. You will then build the whole system on your local machine, which uses the configured remote builder to build aarch64-linux stuff. Afterwards the closure is copied to your target.
2. Similar to (1), you can achieve the same behavior by setting boot.binfmt.emulatedSystems = ["aarch64-linux"]; on your local system and prevent building on the aarch64 machine in the first place. This may actually be faster than letting the remote do the building if the remote is very slow (e.g. RaspberryPi 2/3)
3. Explicitly copy the secrets before initiating the build. If you make sure that the derivation is already available on the remote before starting the build, it will never try to build it since it's already there. You can run nix copy .#nixosConfigurations.yourHost.config.age.rekey.derivation ssh://yourHost whenever necessary.
4. Configure your local machine as a substituter on the target machine, so when the target tries to build the derivation it will instead find the existing derivation on your local machine and copy it automatically.

There may also exist other solutions that I haven't thought about yet.

Also please beware that manual copying can lead to caveats when a rekeyed derivation is rebuild without changing it's hash. This may happen if you choose to use a dummy secret value and later rekeying again to use the true secrets. In that case any existing derivations must be replaced in the store, which agenix-rekey does automatically for your local machine but it can't do that for your target machine since it doesn't know about them. This means in certain rare situations you might need to delete and re-copy the derivation to your remote.

I seem to mostly stick to options 1/2 on my local infrastructure. I'll build locally what I can, and if I need more build power I add the remotes as builders to my local machine.

[oddlama]

@phaer @NyCodeGHG There's been a development in this regard, I've introduced a new local storage mode for rekeyed secrets. When using this new local storage mode, all rekeyed secrets will be stored locally in your flake's repository which removes the requirement to build a derivation at all, meaning there should be no more shenanigans with remote builders or CI/CD. You can refer to the Storage Modes in the readme if you want to try this :)