Support "remote build"-style indexing

[magneticflux-]

I have a low-power device that doesn't have enough RAM or CPU to build a full index, so I tried to run nix-index inside a derivation like so:

pkgs.runCommand "nix-index-db" { nativeBuildInputs = [ pkgs.nix-index ]; } "nix-index --db $out --nixpkgs ${pkgs.path}"

Unfortunately, this fails because nix-index uses nix-env internally, which tries to escape the sandbox:

+ querying available packages
error: querying available packages failed
caused by: nix-env failed with error: nix-env failed with exit status: 1:
error: creating directory '/nix/var': Permission denied

Including strace -f in the builder shows that it looks in the 3 default places if NIX_PATH is unset:

[pid   169] newfstatat(AT_FDCWD, "/homeless-shelter/.nix-defexpr", 0x7fffffffbc70, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/homeless-shelter/.nix-defexpr", 0x7fffffffbc60, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/homeless-shelter", 0x7fffffffbab0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] mkdir("/homeless-shelter", 0777) = -1 EACCES (Permission denied)

[pid   169] access("/nix/var/nix", R_OK|W_OK) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/nix/var/nix/daemon-socket/socket", 0x7fffffffb840, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/nix/var/nix", 0x7fffffffb840, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/homeless-shelter/.local/share/nix/root", 0x7fffffffb840, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/homeless-shelter/.local/share/nix/root", 0x7fffffffb830, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/homeless-shelter/.local/share/nix", 0x7fffffffb680, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/homeless-shelter/.local/share", 0x7fffffffb4d0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/homeless-shelter/.local", 0x7fffffffb320, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/homeless-shelter", 0x7fffffffb170, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] mkdir("/homeless-shelter", 0777) = -1 EACCES (Permission denied)

[pid   169] newfstatat(AT_FDCWD, "/nix/store", {st_mode=S_IFDIR|S_ISVTX|0775, st_size=90, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid   169] newfstatat(AT_FDCWD, "/nix/store/.links", {st_mode=S_IFDIR|0755, st_size=2, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid   169] newfstatat(AT_FDCWD, "/nix/var/nix/profiles", 0x7fffffffb520, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/nix/var/nix", 0x7fffffffb370, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/nix/var", 0x7fffffffb1c0, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
[pid   169] newfstatat(AT_FDCWD, "/nix", {st_mode=S_IFDIR|0755, st_size=3, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid   169] mkdir("/nix/var", 0777)     = -1 EACCES (Permission denied)
[pid   169] write(2, "error: creating directory '/nix/"..., 56) = 56

So let's set that, along with NIX_STATE_DIR to allow it to save some other data: "NIX_PATH=\"nixpkgs=${pkgs.path}\" NIX_STATE_DIR=\"$TMPDIR/nix/var\" nix-index -c 9 --db $out --nixpkgs ${pkgs.path}"

nix-index-db> + generating index: 00000 paths found :: 00099 paths not in binary cache :: 118975 paths in queue
nix-index-db> Error: fetching the references of store path '/nix/store/2q6fdkjwwxm8a3kc9mv7s18ad8vvki53-semver-7.5.4' failed
nix-index-db> Caused by: error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
nix-index-db> Caused by: dns error: failed to lookup address information: Temporary failure in name resolution
nix-index-db> Caused by: failed to lookup address information: Temporary failure in name resolution

Yep, I should have known: the call to nix-env was actually downloading info missing from the local store. I'm not proficient in Nix enough to debug further, but I imagine you'd need to make the "db" derivation depend on the references of each derivation to index.