✨ Improved garbage collection defaults

[jedevc]

Spinning this out of a private discord discussion here.

We inherit buildkit's default gc policies - which is to try and keep cache usage under 10% of available disk.

This is not actually great for a lot of cases. Here's a few dagger scenarios, and what kind of cache policies we might want for these:

• Using dagger on a personal linux machine
  • I want dagger to use quite a lot of space - I use dagger for most of my dev work, so maybe it could use 20% of my disk (on the scale of several hundred gigs of data).
• Using dagger in docker-desktop/lima/etc on a personal mac/windows/linux machine
  • Because this is a VM, there's a disk provisioned for it - that's usually quite small (the docker desktop seems unreasonably so sometimes).
  • I want dagger to use most of the available disk storage (but not all), maybe something like 50% (this works out to mostly the same as the above option - but now there are two dials to turn, the docker desktop disk size dial, and the dagger disk size dial).
• Using dagger in a dedicated CI worker
  • I want dagger to use as much storage as it requires - I want the limit to be something like 80-90%.

At the moment, we just have a flat "10%" for everything - which is pretty bad, especially for the last scenario.

Part of this is related to documentation + usability of configuring cache policy in the first place:

• Document how to configure local cache size #7022
• ✨ dagger cli should have cache subcommands #6260

However, this is still different - we should have a way of making sure that the defaults are good for 90% of users. There's unfortunately no easy way to detect what kind of case we're in.

I think this makes sense more sense in the context of #5583 though. When we start a dagger engine with a driver, we can allow specifying the cache policies for that engine, and potentially automatically pick some reasonable defaults (e.g. based on the presence of GITHUB_ env vars, etc).

[gtzo-anchorage]

Just chiming in to say that this puzzled me a fair bit as a first-time Dagger user looking to implement on dedicated CI runners. A more polished cache control would be extremely useful!

[Scalahansolo]

One thing to call out here.... If running Dagger in K8s, Dagger will try to keep it's memory cache around the requested memory, but has the potential to spike well above that. Our memory limit is set really high to account for spikes, but GC can be psuedo tuned by setting that value.

[marcosnils]

@jedevc given that this has bitten quite some people and it might result in a poor experience when running dagger, WDYT about changing the defaults from 10 to something considerably higher like 75%?

It's somehow odd that the defaults are so low, specially when docker doesn't have any gc mechanism to prune its cache volumes and/or images.

[jedevc]

Yeah, that feels like a reasonable option for the short-term honestly.

Just bumping it up should make for a better general experience - but I still think there's some complexity with how to manage different environments, so that would only address the surface level of this IMO.

[marcosnils]

• but I still think there's some complexity with how to manage different environments, so that would only address the surface level of this IMO.

I totally agree with this. Maybe we can bump the default GC retention % for now and move this discussion to #5583 as you initially suggested?