Format error when talking to newer daemon with older client

[SuperSandro2000]

Describe the bug

I am running repeatedly into the following error when building with a nix 2.18.1 cli and talking to a 2.21.0 daemon.

copying 0 paths...
error: boost::too_many_args: format-string referred to fewer arguments than were passed
error:
       … while reading the response from the build hook

       error: unexpected EOF reading a line

Daemon did not contain any useful logs.

Steps To Reproduce

1. start the nix-daemon with 2.21.0
2. start the build with a cli version 2.18.1
3. potentially involve other versions in the remote builders
4. obverse the error

Expected behavior

It should work.

nix-env --version output

nix-env (Nix) 2.18.1

Additional context

Add any other context about the problem here.

Priorities

Add 👍 to issues you find important.

[alisonjenkins]

Updated my system this morning on MacOS and since then have been unable to use Nix to build anything.

Attempted to rollback the profile to generations before the update but I am still getting this error.

Has the new version changed something in the Nix store that is making breaking the old versions?

EDIT:

So rolling back the system generation on MacOS using:

nix-env --list-generations --profile /nix/var/nix/profiles/system

has rolled back most things but for some reason the Nix daemon is still on 2.21.0:

❯ pgrep -fl nix-daemon                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       <region:eu-west-2>
1084 /nix/store/55na9m4x162ai94y8y4d2jzs7z0hck30-nix-2.21.0/bin/nix-daemon
16256 /nix/store/55na9m4x162ai94y8y4d2jzs7z0hck30-nix-2.21.0/bin/nix-daemon

so it seems that the nix-daemon isn't using the system profile... digging further....

Located the nix-daemon service file at: /Library/LaunchDaemons/org.nixos.nix-daemon.plist.

Doesn't seem to be managed by the Nix store so that explains why it hasn't been rolled back.

It has the content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>EnvironmentVariables</key>
	<dict>
		<key>NIX_SSL_CERT_FILE</key>
		<string>/etc/ssl/certs/ca-certificates.crt</string>
		<key>OBJC_DISABLE_INITIALIZE_FORK_SAFETY</key>
		<string>YES</string>
	</dict>
	<key>KeepAlive</key>
	<true/>
	<key>Label</key>
	<string>org.nixos.nix-daemon</string>
	<key>LowPriorityIO</key>
	<false/>
	<key>ProcessType</key>
	<string>Standard</string>
	<key>ProgramArguments</key>
	<array>
		<string>/bin/sh</string>
		<string>-c</string>
		<string>/bin/wait4path /nix/store/55na9m4x162ai94y8y4d2jzs7z0hck30-nix-2.21.0/bin/nix-daemon &amp;&amp; exec /nix/store/55na9m4x162ai94y8y4d2jzs7z0hck30-nix-2.21.0/bin/nix-daemon</string>
	</array>
	<key>SoftResourceLimits</key>
	<dict>
		<key>NumberOfFiles</key>
		<integer>4096</integer>
	</dict>
</dict>
</plist>

Going to attempt manually rewrite it to a Nix nix-daemon path from an older version.

UPDATE:

So after looking at the nix packages in the nix store I found this: /nix/store/xddsxyp60qq5jmdxb9rkwbvxzxa4lhar-nix-2.19.3/Library/LaunchDaemons/org.nixos.nix-daemon.plist

Which contains a better way to manage the nix-daemon version (wonder why my service was using the wrong version?):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>EnvironmentVariables</key>
    <dict>
      <key>OBJC_DISABLE_INITIALIZE_FORK_SAFETY</key>
      <string>YES</string>
    </dict>
    <key>Label</key>
    <string>org.nixos.nix-daemon</string>
    <key>KeepAlive</key>
    <true/>
    <key>RunAtLoad</key>
    <true/>
    <key>ProgramArguments</key>
    <array>
      <string>/bin/sh</string>
      <string>-c</string>
      <string>/bin/wait4path /nix/var/nix/profiles/default/bin/nix-daemon &amp;&amp; exec /nix/var/nix/profiles/default/bin/nix-daemon</string>
    </array>
    <key>StandardErrorPath</key>
    <string>/var/log/nix-daemon.log</string>
    <key>StandardOutPath</key>
    <string>/dev/null</string>
    <key>SoftResourceLimits</key>
    <dict>
      <key>NumberOfFiles</key>
      <integer>1048576</integer>
    </dict>
  </dict>
</plist>

UPDATE:

After rewriting the service file and rebooting (probably could have just restarted the service) the nix-daemon version is now in sync with the client and I am no longer getting this issue. However on further inspection the default profile that the service file from the nix derivation is using nix-daemon 2.17.0. I am going to rewrite it once more to use the system profile so that it stays in sync with the rest of the system.

UPDATE:

Have rewritten it and everything is in sync as expected now. The final version of the services file I ended up with is:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>EnvironmentVariables</key>
    <dict>
      <key>OBJC_DISABLE_INITIALIZE_FORK_SAFETY</key>
      <string>YES</string>
    </dict>
    <key>Label</key>
    <string>org.nixos.nix-daemon</string>
    <key>KeepAlive</key>
    <true/>
    <key>RunAtLoad</key>
    <true/>
    <key>ProgramArguments</key>
    <array>
      <string>/bin/sh</string>
      <string>-c</string>
      <string>/bin/wait4path /nix/var/nix/profiles/system/sw/bin/nix-daemon &amp;&amp; exec /nix/var/nix/profiles/system/sw/bin/nix-daemon</string>
    </array>
    <key>StandardErrorPath</key>
    <string>/var/log/nix-daemon.log</string>
    <key>StandardOutPath</key>
    <string>/dev/null</string>
    <key>SoftResourceLimits</key>
    <dict>
      <key>NumberOfFiles</key>
      <integer>1048576</integer>
    </dict>
  </dict>
</plist>

UPDATE:

It seems for me that Nix 2.21.0 is just broken. Even when the daemon and the client are both in sync:

❯ nix build '#.packages.aarch64-darwin.app'                                                                                                                                                                          <region:eu-west-2>
warning: Git tree '/Users/ajenkins/git/myproj' is dirty
error: boost::too_many_args: format-string referred to fewer arguments than were passed
error:
       … while reading the response from the build hook

       error: unexpected EOF reading a line

░▒▓   …/myproj   mybranch 🗃️  ×1  v21.7.1   09:49 took 29s
❯ pgrep -fl nix-daemon                                                                                                                                                                                               <region:eu-west-2>
1005 /nix/var/nix/profiles/system/sw/bin/nix-daemon

░▒▓   …/myproj   mybranch 🗃️  ×1  v21.7.1   09:51 
❯ /nix/var/nix/profiles/system/sw/bin/nix-daemon --version                                                                                                                                                           <region:eu-west-2>
nix-daemon (Nix) 2.21.0

░▒▓   …/myproj   mybranch 🗃️  ×1  v21.7.1   09:51 
❯ nix --version                                                                                                                                                                                                      <region:eu-west-2>
nix (Nix) 2.21.0

Going to roll back to a previous version of Nix and pin until this issue is resolved.

[magneticflux-]

I'm having the same issue as @alisonjenkins, my daemon and CLI are both on 2.21.0:

❯ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.7.9, NixOS, 24.05 (Uakari), 24.05.20240313.9dce028`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.21.0`
 - nixpkgs: `/nix/store/c02aa9kbhwm2az5vg2m6vvwzrrj3qdqs-xgviqbk8bf1w47kfyqakqhq92a1v3n8v-source`

I have a few SSH remote builders configured; my /nix/store is on a dedicated ZFS dataset; I'm building flakes with nix build .#xyz.

EDIT:
It seems like the SSH remote builders either cause or amplify the cause of the issue; adding --builders "" to my command to reset the list of remote builders lets it succeed.

EDIT EDIT:
More specifically, it seems like the issue occurs when jobs are simultaneously scheduled for remote builders AND the local builder. My previous edit worked because it prevented all remote builds, but I just found that it also works to specify -j 0 to prevent any local builds from running. Hopefully this gives someone a place to start investigating further, as I'm not familiar with Nix internals. @SuperSandro2000 does this match with what you're experiencing?

[lf-]

presented without comment

149bd63#diff-9add3e94cd9b2ff95883ba6c024b0be4fd6fcaec42f12ff1def80bf68e6f539fR152-R154

nix/src/build-remote/build-remote.cc

Lines 205 to 210 in 290072c

	auto error = HintFmt(errorText);
	error
	% drvstr
	% neededSystem
	% concatStringsSep<StringSet>(", ", requiredFeatures)
	% machines.size();