Lazy trees #6530

edolstra · 2022-05-12T19:10:40Z

Features

Flake inputs are now only copied to the Nix store if that's actually needed (fixing Copy local flakes to the store lazily #3121). That is, an operation like nix build nixpkgs#hello no longer copies the nixpkgs flake to the Nix store. Only an expression like src = ./. or nix.registry.nixpkgs.flake = nixpkgs; will cause an entire flake to be copied.
fetchTree can now apply patches to a tree, e.g. fetchTree { ...; patches = [ ./foo.patch ./bar.patch ]; } (Support flake references to patches #3920). Patches are applied in memory, so this doesn't require the entire tree to be materialized to disk.
The github fetcher now fetches zip archives instead of tarballs, and does not unpack them to disk. Depending on the filesystem, this can save a lot of disk space. (E.g. on ext4 with a 4 KiB block size, an unpacked Nixpkgs tree takes 252 MiB, whereas the zip archive takes 43 MiB.

Design

The evaluator no longer accesses the filesystem directly. Instead, it uses a virtual filesystem abstraction called InputAccessor, which has FS operations like readFile() . You can get the InputAccessor for an Input by calling Input::lazyFetch(), which unlike Input::fetch() does not (necessarily) copy the input to the Nix store.

The following InputAccessors currently exist:

FSInputAccessor: This directly accesses the host filesystem, optionally rooted at some prefix. This is used for both non-flake evaluation (e.g. nix-build ./foo.nix), for path:// flakes, and for (dirty or clean) Git working directories (git+file://). It optionally takes a set of allowed paths, which is used by the Git input type to ensure that you can't access untracked files.
MemoryInputAccessor: An in-memory filesystem, used to provide <nix/fetchurl.nix>.
ZipInputAccessor: An accessor that reads directly from a zipfile, using libzip. Used by the github fetcher.
PatchingInputAccessor: An adaptor that wraps another accessor whose readFile() applies a patch to the file returned by the underlying accessor.

Paths are represented inside the evaluator using the SourcePath type, which is a tuple of an InputAccessor and a path relative to the root of the input. So the SourcePath {input->lazyFetch(), "/flake.nix"} represents the file flake.nix inside some input.

Incompatible changes

The outPath attribute returned by fetchTree et al. is no longer a store path but a path (i.e. a SourcePath). This can cause some breakage, e.g. in NixOS, setting

nix.registry.nixpkgs.flake = nixpkgs;

causes the error A definition for option 'nix.registry.nixpkgs.to.path' is not of type 'string or signed integer or boolean or package'. Instead you have to write:

nix.registry.nixpkgs.flake = "${nixpkgs}";

to force the argument to be a store path.

To do

Fix many test failures.
Fix showing source code in error messages. Currently the error formatter doesn't know anything about SourcePaths.
Give a helpful message when trying to use untracked files in a git repo (Be smart and helpful when some files are missing during the evaluation #4507).
Fix the evaluation cache.
Fix flake locking.
Get rid of the narHash attribute in flake.lock? Computing it is expensive since it requires reading the entire source tree. However, we need it to be able to substitute flake inputs, but maybe we don't care about that.
What to do with path:// inputs? They're only locked by narHash, so if we don't have narHash, we can't lock path:// inputs.
Fix/improve subflake handling.
Update release notes / docs.

Context

Closes Copy local flakes to the store lazily #3121
Closes Support flake references to patches #3920
Closes Restrict builtins.isStorePath #5973

jaen · 2022-05-13T15:11:52Z

Sorry if this is dumb question, but does it have any bearing on support for git LFS and similar features (git-crypt git annex)? I could imagine the virtual file system accessor transparently handling reading the actual large file instead of the pointer file (this is what currently happens for me when I try to use git LFS for my secrets with sops-nix and it breaks the rebuild) to make this work.

Could this be a right use-case for this feature, or am I misunderstanding it?

blaggacao · 2022-05-18T22:10:44Z

What to do with path:// inputs? They're only locked by narHash, so if we don't have narHash, we can't lock path:// inputs. maybe we don't care about that.

I agree. We don't care. This greately improves the UX of relative flakes within a repository.

Absolut paths are probably only sensible as a development helper.

roberth · 2022-05-26T09:43:10Z

Absolut paths are probably only sensible as a development helper.

and the relative paths like subflakes can be locked by locking their parent flake instead.

nixos-discourse · 2022-06-03T09:48:49Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-31/19481/1

layus · 2022-06-03T19:00:25Z

src/libfetchers/zip-input-accessor.cc

+    std::string readFile(const CanonPath & path) override
+    {
+        if (lstat(path).type != tRegular)
+            throw Error("file '%s' is not a regular file");


nit:

-throw Error("file '%s' is not a regular file"); +throw Error("file '%s' is not a regular file", path);

I am getting this error while running update:

nix run github:edolstra/nix?ref=lazy-trees#nix -- flake update --show-trace warning: Git tree '/my/project/dir' is dirty error: file '%s' is not a regular file … while updating the flake input 'mach-nix' … while updating the lock file of flake 'git+file:///my/project/dir?ref=refs%2fheads%2fmy-current-branch'

and mach-nix input is defined like this:

mach-nix.url = "github:DavHau/mach-nix"; mach-nix.inputs.nixpkgs.follows = "nixpkgs"; mach-nix.inputs.flake-utils.follows = "flake-utils";

Very strange, the error is with flake.lock:

error: file '/flake.lock' is not a regular file … while updating the flake input 'mach-nix' … while updating the lock file of flake 'git+file:///my/project/dir?ref=refs%2fheads%2fmy-current-branch'

Oh, mach-nix has a symlink for the flake.lock. This explains the error.

https://github.com/DavHau/mach-nix/blob/master/flake.lock

I guess readFile has to resolve symlinks now. This in turns asks the question of 1) do we want to support that, 2) Should we prevent symlinks that go out of the flake boundary and 3) do we accept that lazy flakes (that live in a zip archive) will have a different behavior regarding symlinks that escape the flake than flakes that live plainly in the store.

@edolstra This patch works for me 🥳 : layus@44d5bd4

I think it is safe because paths are rooted at the archive root. But is it always correct ? Could be.

nrdxp · 2022-06-03T23:10:57Z

These changes seem to disallow a flake from setting a different version of itself as an input via a url. This was allowed before. To reproduce just run nix flake lock github:input-output-hk/cardano-node as an example. The following inputs triggers the problem:
https://github.com/input-output-hk/cardano-node/blob/7ee6467e288aee5021f048a1c898c05881060e4b/flake.nix#L44-L49

Not sure how or why this worked before and not now, just kinda an odd find I thought was worth mentioning.

nixos-discourse · 2022-06-20T13:21:00Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-32/19865/1

icetan · 2022-06-27T13:24:47Z

Wanted to try out relative paths to be able to use subflakes in a practical way, but I ran into some problems.

Root flake file flake.nix:

{
  inputs = {
    utils.url = "github:numtide/flake-utils/bee6a7250dd1b01844a2de7e02e4df7d8a0a206c";
    sub1.url = "path:./sub1";
  };
  outputs = { nixpkgs, utils, sub1, ... }:
    utils.lib.eachDefaultSystem (system: {
      packages = {
        inherit (sub1.packages.${system}) hello;
      };
    });
}

Subflake file sub1/flake.nix:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/9a17f325397d137ac4d219ecbd5c7f15154422f4";
    utils.url = "github:numtide/flake-utils/bee6a7250dd1b01844a2de7e02e4df7d8a0a206c";
  };
  outputs = { nixpkgs, utils, ... }:
    utils.lib.eachDefaultSystem (system:
      let
        legacyPackages = import nixpkgs { inherit system; };
      in
      {
        packages = {
          inherit (legacyPackages) hello;
        };
      });
}

When I run nix flake show with latest from lazy-trees branch:

nix run github:edolstra/nix/04cb555aebfff68732cc7f0120def20449515eea -- flake show --show-trace

I get the output:

path:/home/icetan/src/nix/test-subflakes
└───packages
    ├───aarch64-darwin
error: in pure evaluation mode, 'fetchTree' requires a locked input, at (string):16:18

       … while realising the context of a path

       at «string»:20:19:

           19|
           20|           flake = import (sourceInfo + (if subdir != "" then "/" else "") + subdir + "/flake.nix");
             |                   ^
           21|

       … while evaluating anonymous lambda

       at «string»:10:13:

            9|     builtins.mapAttrs
           10|       (key: node:
             |             ^
           11|         let

       … from call site

       … while evaluating anonymous lambda

       at «string»:23:25:

           22|           inputs = builtins.mapAttrs
           23|             (inputName: inputSpec: allNodes.${resolveInput inputSpec})
             |                         ^
           24|             (node.inputs or {});

       … from call site

I expect something more like when running nix flake show using nix (Nix) 2.9.1:

path:/home/icetan/src/nix/test-subflakes?lastModified=1656334768&narHash=sha256-ODEqelm5y%2fVQkVkX11EWMWOHFpH+8wMZpEySuE0lPWY=
└───packages
    ├───aarch64-darwin
    │   └───hello: package 'hello-2.12'
    ├───aarch64-linux
    │   └───hello: package 'hello-2.12'
    ├───i686-linux
    │   └───hello: package 'hello-2.12'
    ├───x86_64-darwin
    │   └───hello: package 'hello-2.12'
    └───x86_64-linux
        └───hello: package 'hello-2.12'

edolstra · 2022-06-27T16:43:38Z

@icetan Yeah, subflakes are broken, I'll fix it.

nixos-discourse · 2022-08-03T23:08:41Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/running-nixos-rebuild-does-not-add-new-files-have-i-misunderstood-its-purpose/20664/10

nixos-discourse · 2022-08-15T20:51:59Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/import-custom-directories-files-to-package-path-on-nix-store/20991/7

In particular, plain paths no longer ignore query parameters (e.g. '/foo/bar?lock=1').

lheckemann · 2022-08-17T13:34:27Z

src/libcmd/command.cc

 {
+    auto path = file.getPhysicalPath();
+    if (!path)
+        throw Error("cannot open '%s' in an editor because it has no physical path", file);


It would be nice to write it to a temporary file to allow at least read-only operations on the file. By no means a blocker, but maybe a FIXME would be appropriate here?

src/libcmd/command.cc

lheckemann · 2022-08-17T14:09:39Z

src/libcmd/repl.cc

+                if (auto path = std::get_if<SourcePath>(&pos.origin))
+                    return {*path, pos.line};
+                else
+                    throw Error("'%s' cannot be shown in an editor", pos);


This would happen when the position of the lambda can't be determined, right? I think this error message doesn't reflect that very well.

src/libexpr/eval-cache.cc

lheckemann · 2022-08-18T20:34:03Z

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-22.05";
  outputs = {nixpkgs, ...}: {
    defaultPackage.x86_64-linux = nixpkgs.legacyPackages.x86_64-linux.steam;
  };
}

Fails to build because of unfree as expected; however, the source location information in the error message is weird (should probably reference the input name if possible?):

error: Package ‘steam’ in /virtual/4/pkgs/games/steam/steam.nix:43 has an unfree license (‘unfreeRedistributable’), refusing to evaluate.

lheckemann · 2022-08-19T09:10:51Z

One unfortunate consequence of this is that you can no longer just open a path mentioned in an error trace, which can make digging into code for debugging errors a lot more painful. Using --override-input to replace e.g. nixpkgs with a local path can probably help with this, but ideally there would be a way to do this for any input type as it was before lazy-trees.

I can think of several fancy ways to work around this, but they all seem quite expensive and this PR is already a lot bigger than it ideally would be; an option to restore the old behaviour of fetching and unpacking inputs to the store would be extremely helpful here.

Fancy too-big-for-this-PR ideas

nix edit could be expanded to allow opening files within inputs read-only
- For extra fanciness, this could be read-write and Nix could output a patch to apply to the input once the editor is done
I previously mentioned this as a joke, but a FUSE filesystem (nix flake mount?) would provide a much more comfortable input-inspection facility and could also provide the ability to generate patches

When you have a attribute like src = ./.; the root of the flake should produce a store path /nix/store/<hash1>-<hash2>-source, because before lazy trees, the basename of the tree in the store was /nix/store/<hash2>-source. Even though that was really a bug, we do need to mimic this behaviour so we don't change evaluation results.

Issue NixOS#10627.

This never really worked, since the resulting string depends on the location of the source tree (which is only predictable when using flakes) and is not in the context of the derivation (so it's inaccessible when sandboxing is enabled, and not reproducible when sandboxing is disabled). But we don't want to change evaluation results, so warn against it and simulate the pre-lazy-trees behaviour by replacing virtual paths with the store paths that *would* exist if we copied the source tree to the store.

This code caused firstType to be set to nPath rather than nAttrs. That's probably better, but it broke compatibility: https://hydra.nixos.org/build/258397002

This enables caching of repeated hashing of the same path.

This resolves an evaluation error in https://api.flakehub.com/f/pinned/knightpp/modupdate/0.0.8/018d560c-d88c-7c2b-ba5f-918283ea4c47/source.tar.gz, which does something like filter = path: type: builtins.elem (/. + path) [ ./go.mod ... ]; leading to error: a string that refers to a store path cannot be appended to a path The hack is that if we append a virtual path to /., we discard the orginal accessor and use the one from the right-hand side.

adrian-gierakowski · 2024-05-17T07:45:47Z

Is there a list that of known edge-cases and regressions one should be aware of when n case one would like to use this branch? Thank you!

This comment was marked as outdated.

Sign in to view

edolstra mentioned this pull request May 27, 2022

Fix a segfault in the git fetcher #6582

Merged

Atemu mentioned this pull request Jun 3, 2022

Copy local flakes to the store lazily #3121

Open

layus reviewed Jun 3, 2022

View reviewed changes

nrdxp mentioned this pull request Jun 3, 2022

incompatible with pending Nix changes divnix/std#77

Open

This was referenced Jul 5, 2022

Improved flake support hercules-ci/arion#129

Open

nixopsUnstable broke after upgrade recently NixOS/nixpkgs#139532

Closed

roberth mentioned this pull request Jul 18, 2022

New builtin: pathType #3096

Closed

This was referenced Aug 2, 2022

[flake] nix flake init failed with ~/.local/share/nix/root used as a chroot store #6853

Open

Don't ignore unstaged files in local flakes #6858

Open

edolstra added 5 commits August 16, 2022 14:58

nix flake metadata: Don't show "Inputs" if there are no inputs

4adb32f

Support locking path inputs

a218dd8

Merge remote-tracking branch 'origin/master' into lazy-trees

795b21f

Simplify error message

18c6daf

parseFlakeRefWithFragment() improvements

a5fd4ea

In particular, plain paths no longer ignore query parameters (e.g. '/foo/bar?lock=1').

lheckemann reviewed Aug 17, 2022

View reviewed changes

This comment was marked as resolved.

Sign in to view

Merge remote-tracking branch 'origin/master' into lazy-trees

37d859b

edolstra added 19 commits April 30, 2024 16:24

Merge remote-tracking branch 'origin/master' into lazy-trees

db6c219

Debug

a65c8a9

filterPath: Ensure /nix/store/<hash>-<hash>-source

ea54be0

Issue NixOS#10627.

Disable some GitHub narHash related tests for now

a435a40

Run more flake regression tests

7332321

TarballInputScheme: Fix fetchToStore() caching

299d0b7

Merge remote-tracking branch 'origin/master' into lazy-trees

3a96ba0

EvalState: Rename decodePaths() -> prettyPrintPaths()

25635e5

Pretty-print virtual paths in more contexts

b5be3b0

Add tests for tree laziness

2409e6b

Remove unused

4133514

Merge remote-tracking branch 'origin/master' into lazy-trees

2cb003c

Fix concatenating to an attrset

8fc36e2

This code caused firstType to be set to nPath rather than nAttrs. That's probably better, but it broke compatibility: https://hydra.nixos.org/build/258397002

Merge remote-tracking branch 'origin/master' into lazy-trees

bcb1be4

Merge remote-tracking branch 'origin/master' into lazy-trees

f0cf5d8

Merge remote-tracking branch 'origin/master' into lazy-trees

a123f87

Merge remote-tracking branch 'origin/master' into lazy-trees

26f1431

roberth mentioned this pull request May 13, 2024

Virtual path semantics that are deterministic, lazy and suitable for performing deduplication of values by a path value key. #10689

Open

edolstra added 8 commits May 13, 2024 15:12

Merge remote-tracking branch 'origin/master' into lazy-trees

b7255f8

flake-regressions.sh: Make the sort order deterministic

584fece

Merge remote-tracking branch 'origin/master' into lazy-trees

4d39908

builtins.toPath: Rewrite virtual paths

e255be6

Use fetchToStore() instead of computeStorePath()

a33aad3

This enables caching of repeated hashing of the same path.

Warn about double copies

35bbd85

Skip a broken test

b6be10d

traxys mentioned this pull request May 16, 2024

Violation of the flake schema nix-community/nixvim#1525

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lazy trees #6530

Lazy trees #6530

edolstra commented May 12, 2022 •

edited by roberth

jaen commented May 13, 2022

This comment was marked as outdated.

blaggacao commented May 18, 2022

roberth commented May 26, 2022

nixos-discourse commented Jun 3, 2022

layus Jun 3, 2022

layus Jun 9, 2022

layus Jun 9, 2022

layus Jun 9, 2022

layus Jun 9, 2022

nrdxp commented Jun 3, 2022 •

edited

nixos-discourse commented Jun 20, 2022

icetan commented Jun 27, 2022

edolstra commented Jun 27, 2022

nixos-discourse commented Aug 3, 2022

nixos-discourse commented Aug 15, 2022

lheckemann Aug 17, 2022

lheckemann Aug 17, 2022

This comment was marked as resolved.

lheckemann commented Aug 18, 2022

lheckemann commented Aug 19, 2022

adrian-gierakowski commented May 17, 2024

Lazy trees #6530

Are you sure you want to change the base?

Lazy trees #6530

Conversation

edolstra commented May 12, 2022 • edited by roberth

Features

Design

Incompatible changes

To do

Context

jaen commented May 13, 2022

This comment was marked as outdated.

blaggacao commented May 18, 2022

roberth commented May 26, 2022

nixos-discourse commented Jun 3, 2022

layus Jun 3, 2022

Choose a reason for hiding this comment

layus Jun 9, 2022

Choose a reason for hiding this comment

layus Jun 9, 2022

Choose a reason for hiding this comment

layus Jun 9, 2022

Choose a reason for hiding this comment

layus Jun 9, 2022

Choose a reason for hiding this comment

nrdxp commented Jun 3, 2022 • edited

nixos-discourse commented Jun 20, 2022

icetan commented Jun 27, 2022

edolstra commented Jun 27, 2022

nixos-discourse commented Aug 3, 2022

nixos-discourse commented Aug 15, 2022

lheckemann Aug 17, 2022

Choose a reason for hiding this comment

lheckemann Aug 17, 2022

Choose a reason for hiding this comment

This comment was marked as resolved.

lheckemann commented Aug 18, 2022

lheckemann commented Aug 19, 2022

adrian-gierakowski commented May 17, 2024

edolstra commented May 12, 2022 •

edited by roberth

nrdxp commented Jun 3, 2022 •

edited