garbage-collection.md

How does lorri protect project environments from Nix garbage collection?

Most Nix operations never delete packages from the system, they only create new user environments. Garbage collection (GC) is the process by which Nix explicitly removes all packages for which there is no generation, profile, or garbage collector root referencing it.

From the blog post that introduced lorri:

Nix shells are not protected from garbage collection. [...] lorri captures development dependencies and creates garbage collection roots automatically.

What does this mean precisely, and how does it work?

Garbage collection by example: Nix shell and lorri

Let's see GC in action when using Nix shell and lorri.

Example setup

Assume that we have a project directory with the following minimal setup:

$ cat .envrc 
eval "$(lorri direnv)"
$ cat shell.nix 
with import <nixpkgs> {};
mkShell { buildInputs = [ hello ]; }

Nix shell: output dependencies are garbage collected by default

From the Nix manual:

The behaviour of the garbage collector is affected by the keep-derivations (default: true) and keep-outputs (default: false) options in the Nix configuration file. The defaults will ensure that all derivations that are build-time dependencies of garbage collector roots will be kept and that all output paths that are runtime dependencies will be kept as well. All other derivations or paths will be collected. (This is usually what you want, but while you are developing it may make sense to keep outputs to ensure that rebuild times are quick.)

You can check the values of these settings as follows (default values shown here):

$ nix show-config | grep 'keep-\(outputs\|derivations\)'
keep-derivations = true
keep-outputs = false

Nix GC deletes the hello package previously installed via nix-shell if keep-outputs is false (the default):

$ hello
The program ‘hello’ is currently not installed. [...]
$ nix-shell
[...]
copying path '/nix/store/4w99qz14nsahk0s798a5rw5l7qk1zwwf-hello-2.10' from 'https://cache.nixos.org'...
$ hello
Hello, world!
$ exit
$ nix-store --gc
[...]
deleting '/nix/store/4w99qz14nsahk0s798a5rw5l7qk1zwwf-hello-2.10'
deleting '/nix/store/dhmin7wq99aw9f59jm41varj0753va9b-hello-2.10.drv'
deleting '/nix/store/q0282y7l6f59z71hg1pi2v04dfb1jqbl-hello-2.10.tar.gz.drv'
[...]

lorri: output dependencies are protected from garbage collection

With lorri, the hello package is not deleted in a subsequent GC even when keep-outputs is false:

$ hello
The program ‘hello’ is currently not installed. [...]
$ direnv allow # lorri installs `hello` in the background
[...]
$ hello
Hello, world!
$ cd
direnv: unloading
$ nix-store --gc
[...]
0 store paths deleted, 0.00 MiB freed

How does lorri protect dependencies from Nix garbage collection?

In the previous section, we have seen that lorri protects project dependencies from Nix GC. In this section, we will take a closer look at how this is done.

Garbage collection roots

The easy part of protecting a project environment from Nix GC is to create a GC root for it. A GC root is simply a symlink somewhere in /nix/var/nix/gcroots/ that points (directly or indirectly) to a Nix store path. That store path is then protected from GC.

After each successful build, lorri creates an indirect GC root:

• a symlink in $CACHE_DIR/lorri/gc_roots/ (see ProjectDirs::cache_dir for how $CACHE_DIR is determined) which points to the store path of the environment, and
• an indirect Nix GC root in /nix/var/nix/gcroots/per-user/$USER/ which points to the symlink

Here is an example:

$ tree ~/.cache/lorri/gc_roots/
├── [...]
├── 8562d49821e3218f74e0e37413973802
│   └── gc_root
│       └── shell_gc_root -> /nix/store/8pxs717wgd15i8g18v5aqm34icy756ii-lorri-wrapped-project-nix-shell
└── [...]
$ tree /nix/var/nix/gcroots/per-user/leo/
/nix/var/nix/gcroots/per-user/leo/
├── [...]
├── 8562d49821e3218f74e0e37413973802-shell_gc_root -> /home/leo/.cache/lorri/gc_roots/8562d49821e3218f74e0e37413973802/gc_root/shell_gc_root
└── [...]

Why make the GC root indirect?

It makes it easy to garbage collect all lorri-created environments at once: by removing $CACHE_DIR/lorri/gc_roots/. As a result, the garbage collections roots lorri created inside /nix/var/nix/gcroots/ will point nowhere.

The next time Nix GC is triggered, it will fail to follow those GC roots. Roots that can't be followed are deleted. Since the store paths of the environments lorri has created will no longer have any GC roots pointing to them, they will be GC'd.

There are two things to note here:

1. The exact layout and naming of the directories and symlinks can change at any point. They are considered implementation details.
2. The name of the GC root is derived from the file path of the Nix shell file that defines the project environment. As a result, the last successful build of each project environment is protected from Nix GC.

Environment closures

The tricky part of protecting a project environment from Nix GC is to capture all the environment's dependencies in a closure.

The goal of lorri's GC protection mechanism is to keep outputs paths of build-time dependencies for your development environments.

Any package that is a (transitive) runtime dependency of some other package that has a GC root is itself also protected from GC. So to protect build-time dependencies from GC, lorri pretends that they are runtime dependencies.

For Nix, every runtime dependency is a build-time dependency, but not every build-time dependency is a runtime dependency. As an example, gcc is a build-time dependency for many packages but few packages need it at runtime.

The way Nix determines whether a build-time dependency is also a runtime dependency is as follows: once a build has completed, it scans through the build result, looking for the store paths of build-time dependencies. Any build-time dependency whose store path is found in the build result is considered a runtime dependency.

To make Nix believe that all build-time dependencies are also runtime dependencies, lorri exports the value of all environment variables into a file in the output directory of the project closure. This includes PATH as well as environment variables exported by the project environment declaration itself, like buildInputs.

As an example, the environment export file for the example shell.nix from above contains the following line:

$ grep -F 'buildInputs' bash-export
declare -x buildInputs="/nix/store/4w99qz14nsahk0s798a5rw5l7qk1zwwf-hello-2.10"

As a result, Nix will pick up this version of the hello package as a runtime dependency. The GC root for the project environment will thus protect hello from being garbage collected.