Remove shim #7367

sipsma · 2024-05-13T18:56:54Z

This PR gets rid of our shim and migrates all of its functionality to run directly in the engine instead.

Original motivation: as part of the general engine refactoring/cleanup I've been working through I realized we would be able to simplify a lot if we served nested clients (e.g. module functions and nested execs) directly from our executor. Among other things, it would allow us to be more "stateless" and not need to pre-register nested clients (which is a confusing and bug prone set of code paths).

But at that point it felt like the right time to just bite the bullet and rm the shim entirely, which is possible now thanks to our new worker/executor and has a ton of other side benefits, e.g.

Better performance
- starting a container only requires starting runc, not "outer shim"->runc->"inner shim"
- service health checks and tunneling don't need to start a container at all anymore. They only need a raw network namespace, not a full container (https://github.com/dagger/dagger/pull/7367/files#diff-1f2692364baf2a0e130fb4ec0eefc7a170e52149b1beb5114dd4e8a9feec0dfcR68)
- nested clients will be able to connect direct over http to the engine, no grpc tunneling required
- I don't believe these are bottlenecks at our current stage in a lot of cases, but they become more prominent in use cases that result in lots of short execs being spun up and/or a ton of data flowing back and forth between modules+engine.
More debuggability
- right now when the shim has a problem, it can be very hard to debug; I often resort to returning special exit codes to figure out what's going wrong, which is about as low fidelity as it gets. Now it's all just in the engine process so we can just log as needed.
Simplicity
- don't need to smuggle information to the shim through special env vars (and worry about cache invalidation, etc.)
Easier customization
- switching to a non-runc runtime (like containerd) won't involve hacks to shove the shim into the equation somewhere
Security
- nested clients won't have access to the whole engine socket; they only have access to make calls in their session
Various other smaller benefits like avoiding problems with overriding user exec init processes, etc.

Status: I've migrated everything now, just some cleanup and testing left:

double check ONLCR stuff by running vim in an interactive terminal with line numbers
- still can't replicate any problems with this, so my best guess until proven otherwise is that the more recent change to use the CLI's raw terminal rather than going through an emulator invalidated the need for it? It makes sense on some level in that I don't see other tools built on top of the same containerd/runc stack, such as docker, needing to change ONLCR (afaik anyways) and those work interactively. cc @vito
double check otel live exporting w/ bass
double check healthcheck/tunneling logs usage of GlobalLogger, see if output can look same as before for clients
- improved here: f3a48c7 should look the same as before now
see whether our lack of an automatic init in each container is a meaningfully breaking change
- I think this is a meaningful issue; I also noticed now that the python -m http.server service we use in tests no longer responds to SIGTERM (though it does respond to SIGINT). This explains why and how it's related to the fact that we now allow user processes to be init.
- Going with packaging dumb-init into the engine container and injecting it as the init for all containers for now in order to avoid this being a breaking change: c99840a
- This approach should avoid all the weird unexpected behavior of not having an init in the container while also:
  - being extremely lightweight; dumb-init is 20kb statically linked and extremely simple (hence the name)
  - not being strictly required; we can easily add an option to the container API to disable this for use cases that require the user exec is the init (e.g. testing systemd, etc.)
manually test dagger call with local dirs; I saw some unexpected error at one point that could be related to filesync tweaks but don't see this in integ tests so far, so worth a manual double check
- fixed that here 130e79f

I will of course also cleanup the commit history here with more thorough explanations of the changes.

sipsma · 2024-05-17T01:43:02Z

Tests are passing and double checked everything I want to manually double check, so this should be good to start reviewing cc @jedevc @vito.

There's a couple TODOs left about some simplifications made possible by commits later in the series and areas in need of more commenting, but not expecting any significant changes left before feedback.

jedevc

Ahhh this is neat 🎉 Really nice to have the shim gone - even just from an architectural perspective, the practical performance implications are super great as well. Agreed on the potential containerd-worker stuff as well - since containerd already has it's own shim layer for calling runc, having another layer there was horrible.

Despite the large diff, it's pretty easy to read - most of the code is just moved, and the big refactorings are mostly fairly easy to understand. One big thing to call out though - we definitely need to make sure to test the new Filesyncer on Windows, there's a lot of fragile stuff there (ahhh we need tests for this #3462).

I'm not as familiar with the services/networking here, so not really able to comment on how that's changed - I like the RunInNetNS function though, that's neat.

I will of course also cleanup the commit history here with more thorough explanations of the changes.

This would definitely be appreciated ❤️ If you do this, merging without a squash would also be great, that way we can have a bit more insight with blame 🎉

core/containerExec.go

core/schema/modulesource.go

engine/client/client.go

jedevc · 2024-05-17T11:44:19Z

engine/client/filesync.go

+	if rootDir == "" {
+		rootDir = "/"
+	}


We probably want a different default on windows 🪟 See my previous comment as well about this, there's some complexity here.

Specifically, I think we probably want to have this be empty, and not a drive letter. But then this also means that the abs path returned might not begin with a /, bleh (again, see above).

Right, did you end up with a good way of testing dagger on windows? I'll just fallback to running a VM on my laptop, but double checking there's not some easy way to run tests on a GHA windows runner manually or something.

(I just installed it in a VM, it was less painful than I remember other than the fact that I can't run nested docker in there and have to connect to a remote docker on linux over tailscale 🤪)

Okay fixed this here: 48e774a

Tested on windows manually and could still call modules loaded from local sources, so I think we are good there.

Let me know if you see anything off still, obviously manual testing is prone to missing cases.

end up with a good way of testing dagger on windows

Nope :( I just ended up using my personal windows install that I have around for gaming.

Will try and find another moment to give it a go 🎉

jedevc · 2024-05-17T12:15:49Z

engine/buildkit/executorSpec.go

I like this approach, it makes the executor significantly easier to understand, good refactoring 🎉

Of course, now I'm imagining how these callbacks could operate on a [T any] type, and then potentially upstream into buildkit? There's not a lot of dagger-specific logic left in executor.go.

Maybe again I'm misreading it, but suddenly it looks a lot more doable than when we discussed it previously in #7228 (comment), now the hook mechanism here contains almost all of the complexity - and is so, so, much more readable.

For context - I promise I'm not trying to be super nitpicky here 😅 The reason I'm keen on not having a hard-fork of these files as much is because we've definitely had some horrible context-lifetime bugs, etc with runc in callWithIO. These were really nasty to track down and understand, and I'm just trying to come up with ways to avoid divergence for some of those code paths.

Totally, it's much more doable than before, though I think it might require refactoring upstream to use the same sort of approach. Honestly the simplest thing we could do upstream might just be to capitalize some of the structs/funcs it uses so we can import them and use them?

I think it might require refactoring upstream to use the same sort of approach

Yup, I think that's what I was imagining - the upstream implementation as is is quite complex, I think this approach is much easier to read and understand, so it definitely has my vote :)

vito

A mic drop of a PR if I've ever seen one 🎉

vito · 2024-05-17T22:35:42Z

core/c2h.go

-
-	for _, port := range d.tunnelServicePorts {
-		var frontend int
+	getFrontend := func(port PortForward) int {


super nit: might make sense as a method, if there's a clear name. (FrontendOrBackend? FrontOrBackEnd? End? lol)

I went to do this and just saw FrontendOrBackend already exists haha: https://github.com/sipsma/dagger/blob/48e774a1e499482d67ae8c05760663b951751c95/core/net.go#L81-L81

Updating to that 😄

vito · 2024-05-17T22:50:05Z

engine/buildkit/client.go

+	return c.worker.newNetNS(ctx, hostname)
+}
+
+func RunInNetNS[T any](


vito · 2024-05-17T22:53:50Z

core/service.go

+	// we don't need a full container, just a CNI provisioned network namespace to listen in
+	netNS, err := bk.NewNetworkNamespace(ctx, fullHost)


nice nice, much more svelte

core/containerExec.go

vito · 2024-05-17T22:58:17Z

core/healthcheck.go

+	for _, port := range ports {
+		retry := backoff.NewExponentialBackOff(backoff.WithInitialInterval(100 * time.Millisecond))
+		endpoint, err := backoff.RetryWithData(func() (string, error) {
+			return buildkit.RunInNetNS(ctx, d.bk, d.ns, func() (string, error) {


container.go is understandably a pretty big file but the exec related code is the most complicated and interesting part, which makes it annoying to always have to find amongst everything else. Hence, it feels a bit nicer to have in its own dedicated file. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

we only sometimes need the server ID as an env var in order to invalidate cache, so it's a lot cleaner+nicer to just pass it through the exec metadata than rely solely on an env var to pass it around. Also makes it easier for other code (i.e. service + dockerfile frontend) to set it without jumping through the hoops of stringly-typed env vars. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

Not much of note here, just update the OCI spec from the executor rather than the shim. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

Signed-off-by: Erik Sipsma <erik@sipsma.dev>

I was unable to replicate any problems with linebreaks not displaying correctly once this was removed from the shim, even though I verified that stty showed `-onlcr` in `dagger call ... terminal` containers. I *suspect* that the more recent move in the CLI to just use the raw terminal rather than an emulator results in the ONLCR of *my* terminal (as a client) mattering rather than that of the container. Worst case and this is needed after all, we can use OCI hooks for it instead of the shim. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

This is mostly just a transplant of existing code for writing to the meta mount from the shim to the executor. Things worth noting: 1. Secret scrubbing code is a bit simplified by just using filepaths rather than trying to use an fs.FS to support the unit tests. There's less cases to handle and thus less tests needed. The unit tests just rely on t.Tmpdir now, which is just as good for this use case. 2. The `cleanups` util here exists because `defer` doesn't work as expected when setup is spread across multiple functions. It operates the same sort of way but lets you "defer the defer" (if you will) so it only runs when the outermost caller decides it should run. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

The healthchecks and tunneling that used to require a full-blown container that short-circuited to special shim commands can now just be done in the engine directly. That is done here by adding a util for executing go funcs inside the network namespace of existing containers (or creating a new raw network namespace in the case of the tunelling, since we just need an ip and listener there. The util hides the tricky details of needing to lock goroutines so callers can just run the func they want. It works by maintaining a pool of goroutines that will Setns into the container's network namespace when a job comes in over a channel. It also will stay locked in the namespace up to a timeout (currently just a second). * The reasoning here is that we don't want to consume an entire OS thread for too long but also want to avoid rapidly entering and leaving the namespace for e.g. health checks. The OTEL related code is grouped in here since the bulk of the work is setting up the TCP proxying of the otel unix sock, which relies on the above mechanisms. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

This is the last step needed to remove the shim. Things worth noting: * Serving the session is just a matter of getting a listener in the container's network namespace and then using the existing `DAGGER_SESSION_PORT` mechanism (supported by the CLI and all SDKs) to tell the nested client where to make requests to. * There's a slight adjustment to the BuildkitController.Session methods to support directly serving a net.Conn since nested clients no longer need to tunnel through grpc and thus don't need to hit the .Session method directly. * The filesync session attachable was theoretically the trickiest part, but managed to avoid the need to do weird things with trying to use the container's /proc/<pid>/root magic symlink (which is vulnerable to pid-reuse attacks and finicky in that any code that calls filepath.EvalSymlinks on it will break). * Instead, we just do all the bind+overlays mounts to the rootfs rather than relying on runc to do it. We were already doing this for the / mount; we just do it for the rest of the non-special (i.e. not proc, sys, etc.) mounts now too. This gives us a view of the container's filesystem that's read-writable both outside and inside the container. * The other session attachables involving sockets and network tunnelling were more straightforward since they could just re-use the support for running callbacks in network namespaces added in previous commits. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

The removal of the shim was actually a breaking change due to subtle but important differences in how Linux treats pid 1 processes. Namely, pid 1 will not get any default signal handlers and instead must explicitly register them. For example, without the shim our integ tests that ran python -m http.server stopped responding to SIGTERM. Not having an init also made it easier for zombie processes to lurk while the container was still running. To avoid this, we now build dumb-init and package it into the engine container. It's extremely lightweight (20kb statically linked and extremely minimal in what it does) and is also possible make a configuration option on the Container API for users who don't want it (though it should remain the default). This is as opposed to our shim which was much heavier and also was not optional until its functionality migrated to the executor. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

Buildkit frontend implementations sometimes end up creating containers with no network enabled at all, which wasn't handled well previously. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

The executor does not Setns into the mount namespace of containers, only the network namespace, so the Go resolver doesn't have access to the /etc/resolv.conf with search domains specific to the container, which resulted in service hostnames failing to be resolved. While adding support for mount namespace entering is feasible, it would require we do a dance to pre-create a mount namespace and set it in the spec. It's a lot simpler to just check the search domain directly in our code. If any future use cases call for entering mount namespaces we can revisit and just use that instead. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

Buildkit puts these emulators under /dev, but we don't bind mount /dev when setting up the rootfs, so we were ending up binding the emulator to the rootfs but then hiding it when runc mounted /dev. We don't really need to bind mount these ahead of time since we never need to read/write them outside the container, so we can just leave them as bind mounts in the oci spec. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

The previous code assumed that / was universal, which is obviously not the case on Windows. The new code separates out handling a bit cleaner so that Windows clients (i.e. running in the CLI) are handled correctly. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

A few misc cleanups that were a pain to squash into the commit history: * Use consistent file names * Remove outdated comment about meta mount shim permissions (the meta mount doesn't actually get mounted into the final container anymore, it's just an LLB mount so its cache key is tied to the exec op). * Reuse existing method for getting port rather than duplicating. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

sipsma · 2024-05-18T01:52:02Z

This would definitely be appreciated ❤️ If you do this, merging without a squash would also be great, that way we can have a bit more insight with blame 🎉

Done! Did some consolidation to cut the number of commits in half and added more context to the messages as needed.

Hopefully I didn't fat finger anything while resolving conflicts during it. I'll double check with fresh eyes on Monday before pressing merge (will avoid the squash in this case too like you said).

sipsma · 2024-05-20T17:51:39Z

Double-checked the diff after doing commit cleanup, everything looks correct, so I'm gonna do rebase+merge now while it's still up to date with main to get the big diff out of the way for any other pending PRs w/ conflicts. cc @jedevc if you find anything off with Windows still let me know and will fixup in a follow-up!

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [aquaproj/aqua-registry](https://togithub.com/aquaproj/aqua-registry) | minor | `v4.185.1` -> `v4.188.0` | | [casey/just](https://togithub.com/casey/just) | minor | `1.26.0` -> `1.27.0` | | [charmbracelet/gum](https://togithub.com/charmbracelet/gum) | patch | `v0.14.0` -> `v0.14.1` | | [cli/cli](https://togithub.com/cli/cli) | minor | `v2.49.2` -> `v2.50.0` | | [dagger/dagger](https://togithub.com/dagger/dagger) | patch | `v0.11.4` -> `v0.11.6` | | [dprint/dprint](https://togithub.com/dprint/dprint) | minor | `0.45.1` -> `0.46.1` | | [fujiwara/awslim](https://togithub.com/fujiwara/awslim) | patch | `v0.1.1` -> `v0.1.2` | | [golangci/golangci-lint](https://togithub.com/golangci/golangci-lint) | minor | `v1.58.2` -> `v1.59.0` | | [simulot/immich-go](https://togithub.com/simulot/immich-go) | minor | `0.14.1` -> `0.15.0` | | [snyk/cli](https://togithub.com/snyk/cli) | patch | `v1.1291.0` -> `v1.1291.1` | | [twpayne/chezmoi](https://togithub.com/twpayne/chezmoi) | patch | `v2.48.1` -> `v2.48.2` | | [weaveworks/eksctl](https://togithub.com/weaveworks/eksctl) | minor | `v0.179.0` -> `v0.180.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>aquaproj/aqua-registry (aquaproj/aqua-registry)</summary> ### [`v4.188.0`](https://togithub.com/aquaproj/aqua-registry/releases/tag/v4.188.0) [Compare Source](https://togithub.com/aquaproj/aqua-registry/compare/v4.187.1...v4.188.0) [Issues](https://togithub.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.188.0) | [Pull Requests](https://togithub.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.188.0) | aquaproj/aqua-registry@v4.187.1...v4.188.0 #### 🎉 New Packages [#23408](https://togithub.com/aquaproj/aqua-registry/issues/23408) [getsavvyinc/savvy-cli](https://togithub.com/getsavvyinc/savvy-cli): Create, share, and run runbooks from your terminal [#23384](https://togithub.com/aquaproj/aqua-registry/issues/23384) [joshmedeski/sesh](https://togithub.com/joshmedeski/sesh): Smart session manager for the terminal [@CrystalMethod](https://togithub.com/CrystalMethod) ### [`v4.187.1`](https://togithub.com/aquaproj/aqua-registry/releases/tag/v4.187.1) [Compare Source](https://togithub.com/aquaproj/aqua-registry/compare/v4.187.0...v4.187.1) [Issues](https://togithub.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.187.1) | [Pull Requests](https://togithub.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.187.1) | aquaproj/aqua-registry@v4.187.0...v4.187.1 #### Fixes [#23378](https://togithub.com/aquaproj/aqua-registry/issues/23378) authzed/zed: Use gnu binary instead of musl binary [@sapphi-red](https://togithub.com/sapphi-red) musl version exists for linux, but it is dynamically linked to musl libc and therefore doesn't work on glibc systems. ### [`v4.187.0`](https://togithub.com/aquaproj/aqua-registry/releases/tag/v4.187.0) [Compare Source](https://togithub.com/aquaproj/aqua-registry/compare/v4.186.0...v4.187.0) [Issues](https://togithub.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.187.0) | [Pull Requests](https://togithub.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.187.0) | aquaproj/aqua-registry@v4.186.0...v4.187.0 #### 🎉 New Packages [#23375](https://togithub.com/aquaproj/aqua-registry/issues/23375) [authzed/zed](https://togithub.com/authzed/zed): Official command-line tool for managing SpiceDB [@sapphi-red](https://togithub.com/sapphi-red) [#23277](https://togithub.com/aquaproj/aqua-registry/issues/23277) [nucleuscloud/neosync](https://togithub.com/nucleuscloud/neosync): Open source data anonymization and synthetic data orchestration for developers. Create high fidelity synthetic data and sync it across your environments ### [`v4.186.0`](https://togithub.com/aquaproj/aqua-registry/releases/tag/v4.186.0) [Compare Source](https://togithub.com/aquaproj/aqua-registry/compare/v4.185.1...v4.186.0) [Issues](https://togithub.com/aquaproj/aqua-registry/issues?q=is%3Aissue+milestone%3Av4.186.0) | [Pull Requests](https://togithub.com/aquaproj/aqua-registry/pulls?q=is%3Apr+milestone%3Av4.186.0) | aquaproj/aqua-registry@v4.185.1...v4.186.0 #### 🎉 New Packages [#23253](https://togithub.com/aquaproj/aqua-registry/issues/23253) [stripe/stripe-cli](https://togithub.com/stripe/stripe-cli): A command-line tool for Stripe #### Fix [#23254](https://togithub.com/aquaproj/aqua-registry/issues/23254) dlvhdr/gh-dash: Follow up changes of gh-dash v4.0.0 - [dlvhdr/gh-dash#336 </details> <details> <summary>casey/just (casey/just)</summary> ### [`v1.27.0`](https://togithub.com/casey/just/blob/HEAD/CHANGELOG.md#1270---2024-05-25) [Compare Source](https://togithub.com/casey/just/compare/1.26.0...1.27.0) ##### Changed - Use cache dir for temporary files ([#2067](https://togithub.com/casey/just/pull/2067)) ##### Added - Add `[doc]` attribute to set and suppress documentation comments ([#2050](https://togithub.com/casey/just/pull/2050) by [neunenak](https://togithub.com/neunenak)) - Add source_file() and source_directory() functions ([#2088](https://togithub.com/casey/just/pull/2088)) - Add recipe groups ([#1842](https://togithub.com/casey/just/pull/1842) by [neunenak](https://togithub.com/neunenak)) - Add shell() function for running external commands ([#2047](https://togithub.com/casey/just/pull/2047) by [gyreas](https://togithub.com/gyreas)) - Add `--global-justfile` flag ([#1846](https://togithub.com/casey/just/pull/1846) by [neunenak](https://togithub.com/neunenak)) - Add shell-expanded strings ([#2055](https://togithub.com/casey/just/pull/2055)) - Add `encode_uri_component` function ([#2052](https://togithub.com/casey/just/pull/2052) by [laniakea64](https://togithub.com/laniakea64)) - Add `choose` function for generating random strings ([#2049](https://togithub.com/casey/just/pull/2049) by [laniakea64](https://togithub.com/laniakea64)) - Add predefined constants ([#2054](https://togithub.com/casey/just/pull/2054)) - Allow setting some command-line options with environment variables ([#2044](https://togithub.com/casey/just/pull/2044) by [neunenak](https://togithub.com/neunenak)) - Add prepend() function ([#2045](https://togithub.com/casey/just/pull/2045) by [gyreas](https://togithub.com/gyreas)) - Add append() function ([#2046](https://togithub.com/casey/just/pull/2046) by [gyreas](https://togithub.com/gyreas)) - Add --man subcommand ([#2041](https://togithub.com/casey/just/pull/2041)) - Make `dotenv-path` relative to working directory ([#2040](https://togithub.com/casey/just/pull/2040)) - Add `assert` expression ([#1845](https://togithub.com/casey/just/pull/1845) by [de1iza](https://togithub.com/de1iza)) - Add 'allow-duplicate-variables' setting ([#1922](https://togithub.com/casey/just/pull/1922) by [Mijago](https://togithub.com/Mijago)) ##### Fixed - List modules in source order with `--unsorted` ([#2085](https://togithub.com/casey/just/pull/2085)) - Show submodule recipes in --choose ([#2069](https://togithub.com/casey/just/pull/2069)) - Allow multiple imports of the same file in different modules ([#2065](https://togithub.com/casey/just/pull/2065)) - Fix submodule recipe listing indentation ([#2063](https://togithub.com/casey/just/pull/2063)) - Pass command as first argument to `shell` ([#2061](https://togithub.com/casey/just/pull/2061)) - Allow shell expanded strings in mod and import paths ([#2059](https://togithub.com/casey/just/pull/2059)) - Run imported recipes in root justfile with correct working directory ([#2056](https://togithub.com/casey/just/pull/2056)) - Fix output `\r\n` stripping ([#2035](https://togithub.com/casey/just/pull/2035)) ##### Misc - Forbid whitespace in shell-expanded string prefixes ([#2083](https://togithub.com/casey/just/pull/2083)) - Add Debian and Ubuntu install instructions to readme ([#2072](https://togithub.com/casey/just/pull/2072)) - Remove snap installation instructions from readme ([#2070](https://togithub.com/casey/just/pull/2070)) - Fallback to wget in install script if curl isn't available([#1913](https://togithub.com/casey/just/pull/1913) by [tgross35](https://togithub.com/tgross35)) - Use std::io::IsTerminal instead of atty crate ([#2066](https://togithub.com/casey/just/pull/2066)) - Improve `shell()` documentation ([#2060](https://togithub.com/casey/just/pull/2060) by [laniakea64](https://togithub.com/laniakea64)) - Add bash completion for snap ([#2058](https://togithub.com/casey/just/pull/2058) by [albertodonato](https://togithub.com/albertodonato)) - Refactor list subcommand ([#2062](https://togithub.com/casey/just/pull/2062)) - Document working directory ([#2053](https://togithub.com/casey/just/pull/2053)) - Replace FunctionContext with Evaluator ([#2048](https://togithub.com/casey/just/pull/2048)) - Update clap to version 4 ([#1924](https://togithub.com/casey/just/pull/1924) by [poliorcetics](https://togithub.com/poliorcetics)) - Cleanup ([#2026](https://togithub.com/casey/just/pull/2026) by [adamnemecek](https://togithub.com/adamnemecek)) - Increase --list maximum alignable width from 30 to 50 ([#2039](https://togithub.com/casey/just/pull/2039)) - Document using `env -S` ([#2038](https://togithub.com/casey/just/pull/2038)) - Update line continuation documentation ([#1998](https://togithub.com/casey/just/pull/1998) by [laniakea64](https://togithub.com/laniakea64)) - Add example using GNU parallel to run tasks in concurrently ([#1915](https://togithub.com/casey/just/pull/1915) by [amarao](https://togithub.com/amarao)) - Placate clippy: use `clone_into` ([#2037](https://togithub.com/casey/just/pull/2037)) - Use --command-color when printing shebang recipe commands ([#1911](https://togithub.com/casey/just/pull/1911) by [avi-cenna](https://togithub.com/avi-cenna)) - Document how to use watchexec to re-run recipes when files change ([#2036](https://togithub.com/casey/just/pull/2036)) - Update VS Code extensions in readme ([#2034](https://togithub.com/casey/just/pull/2034)) - Add rust:just repology package table to readme ([#2032](https://togithub.com/casey/just/pull/2032)) </details> <details> <summary>charmbracelet/gum (charmbracelet/gum)</summary> ### [`v0.14.1`](https://togithub.com/charmbracelet/gum/releases/tag/v0.14.1) [Compare Source](https://togithub.com/charmbracelet/gum/compare/v0.14.0...v0.14.1) #### What's Changed - Show help with Huh? by [@maaslalani](https://togithub.com/maaslalani) in [charmbracelet/gum#587 - Support using the Home/End keys in pager by [@lzm0](https://togithub.com/lzm0) in [charmbracelet/gum#548 ##### Fixes - Handle `huh?` user aborted error by [@MikaelFangel](https://togithub.com/MikaelFangel) in [charmbracelet/gum#578 - Cursor styling by [@MikaelFangel](https://togithub.com/MikaelFangel) in [charmbracelet/gum#592 - Re-introduce <kbd>ESC</kbd> for `gum write` by [@MikaelFangel](https://togithub.com/MikaelFangel) in [charmbracelet/gum#579 #### New Contributors - [@CodeZea1ot](https://togithub.com/CodeZea1ot) made their first contribution in [charmbracelet/gum#542 - [@camcui](https://togithub.com/camcui) made their first contribution in [charmbracelet/gum#537 - [@lzm0](https://togithub.com/lzm0) made their first contribution in [charmbracelet/gum#548 **Full Changelog**: charmbracelet/gum@v0.14.0...v0.14.1 *** <a href="https://charm.sh/"><img alt="The Charm logo" src="https://stuff.charm.sh/charm-badge.jpg" width="400"></a> Thoughts? Questions? We love hearing from you. Feel free to reach out on [Twitter](https://twitter.com/charmcli), [The Fediverse](https://mastodon.technology/@charm), or [Slack](https://charm.sh/slack). </details> <details> <summary>cli/cli (cli/cli)</summary> ### [`v2.50.0`](https://togithub.com/cli/cli/releases/tag/v2.50.0): GitHub CLI 2.50.0 [Compare Source](https://togithub.com/cli/cli/compare/v2.49.2...v2.50.0) #### What's Changed - Refactor git credential flow code by [@williammartin](https://togithub.com/williammartin) in [cli/cli#9089 - feat: add json output for `gh pr checks` by [@nobe4](https://togithub.com/nobe4) in [cli/cli#9079 - Rework first auth tests with new gitcredential abstractions by [@williammartin](https://togithub.com/williammartin) in [cli/cli#9095 - list the various alias permutations for the command and subcommands, via '--help' and 'gh reference' by [@gabemontero](https://togithub.com/gabemontero) in [cli/cli#8824 - Removed tty message when checking for extension upgrades by [@leevic31](https://togithub.com/leevic31) in [cli/cli#9088 - Fix doc bug for gh run watch by [@jasonodonnell](https://togithub.com/jasonodonnell) in [cli/cli#9052 - feat: add support for stateReason in `gh pr view` by [@nobe4](https://togithub.com/nobe4) in [cli/cli#9080 - fix: rename the `Attempts` field to `Attempt`; expose in `gh run view` and `gh run ls` by [@cawfeecake](https://togithub.com/cawfeecake) in [cli/cli#8905 - Update regex in changedFilesNames to handle quoted paths by [@anda3](https://togithub.com/anda3) in [cli/cli#9115 - Add a `gh variable get FOO` command by [@arnested](https://togithub.com/arnested) in [cli/cli#9106 - Add macOS pkg installer to deployment ([#7554](https://togithub.com/cli/cli/issues/7554)) by [@paulober](https://togithub.com/paulober) in [cli/cli#7555 - Add integration tests for `gh attestation verify` shared workflow use case by [@malancas](https://togithub.com/malancas) in [cli/cli#9107 - Add build provenance for gh CLI releases by [@malancas](https://togithub.com/malancas) in [cli/cli#9087 - build(deps): bump github.com/gabriel-vasile/mimetype from 1.4.3 to 1.4.4 by [@dependabot](https://togithub.com/dependabot) in [cli/cli#9124 - Build completions during release on macos by [@williammartin](https://togithub.com/williammartin) in [cli/cli#9136 - Clarify Mac OS Installer packages are unsigned by [@andyfeller](https://togithub.com/andyfeller) in [cli/cli#9140 #### New Contributors - [@gabemontero](https://togithub.com/gabemontero) made their first contribution in [cli/cli#8824 - [@jasonodonnell](https://togithub.com/jasonodonnell) made their first contribution in [cli/cli#9052 - [@anda3](https://togithub.com/anda3) made their first contribution in [cli/cli#9115 - [@arnested](https://togithub.com/arnested) made their first contribution in [cli/cli#9106 - [@paulober](https://togithub.com/paulober) made their first contribution in [cli/cli#7555 **Full Changelog**: cli/cli@v2.49.2...v2.50.0 </details> <details> <summary>dagger/dagger (dagger/dagger)</summary> ### [`v0.11.6`](https://togithub.com/dagger/dagger/blob/HEAD/CHANGELOG.md#v0116---2024-05-30) [Compare Source](https://togithub.com/dagger/dagger/compare/v0.11.5...v0.11.6) ##### Added - Add `withName` method to File by [@TomChv](https://togithub.com/TomChv) in [dagger/dagger#7491 ##### Fixed - cli: don't validate flags when requesting `--help` by [@helderco](https://togithub.com/helderco) in [dagger/dagger#7417 - fix container init being wrong platform in arm image by [@sipsma](https://togithub.com/sipsma) in [dagger/dagger#7497 - fix container DNS resolution when host has no search domains by [@sipsma](https://togithub.com/sipsma) in [dagger/dagger#7501 - honor system proxy config during git operations by [@sipsma](https://togithub.com/sipsma) in [dagger/dagger#7504 - fix windows-style paths used as file+directory arg values in dagger call by [@sipsma](https://togithub.com/sipsma) in [dagger/dagger#7506 ##### What to do next? - Read the [documentation](https://docs.dagger.io) - Join our [Discord server](https://discord.gg/dagger-io) - Follow us on [Twitter](https://twitter.com/dagger_io) ### [`v0.11.5`](https://togithub.com/dagger/dagger/blob/HEAD/CHANGELOG.md#v0115---2024-05-27) [Compare Source](https://togithub.com/dagger/dagger/compare/v0.11.4...v0.11.5) ##### Added - cli: `dagger login` cloud traces support by [@aluzzardi](https://togithub.com/aluzzardi) in [dagger/dagger#7125 - cli: improved `--progress=plain` implementation for better visibility by [@jedevc](https://togithub.com/jedevc) in [dagger/dagger#7272 ##### Changed - cli: cleaner tty progress view by [@jedevc](https://togithub.com/jedevc) in [dagger/dagger#7347 [dagger/dagger#7371 [dagger/dagger#7386 - cli: don't show functions that can't be called by [@helderco](https://togithub.com/helderco) in [dagger/dagger#7418 - cli: don't show inherited flags in function commands by [@helderco](https://togithub.com/helderco) in [dagger/dagger#7419 - core: remove shim and switch to [dumb-init](https://togithub.com/Yelp/dumb-init) by [@sipsma](https://togithub.com/sipsma) in [dagger/dagger#7367 ##### Fixed - core: fixed custom CA certs in modules by [@sipsma](https://togithub.com/sipsma) in [dagger/dagger#7356 - cli: don't validate flags when requesting `--help` by [@helderco](https://togithub.com/helderco) in [dagger/dagger#7417 ##### What to do next? - Read the [documentation](https://docs.dagger.io) - Join our [Discord server](https://discord.gg/dagger-io) - Follow us on [Twitter](https://twitter.com/dagger_io) </details> <details> <summary>dprint/dprint (dprint/dprint)</summary> ### [`v0.46.1`](https://togithub.com/dprint/dprint/releases/tag/0.46.1) [Compare Source](https://togithub.com/dprint/dprint/compare/0.46.0...0.46.1) #### Changes - fix: dprint 0.46 release for `cargo install` without `--locked` flag ([#852](https://togithub.com/dprint/dprint/issues/852)) - fix: upgrade to rustls 0.23.8 ([#853](https://togithub.com/dprint/dprint/issues/853)) Please run `dprint config update` after upgrading as some BOM handling happens in the plugins now and some may fail to parse files with a BOM now. See [#854](https://togithub.com/dprint/dprint/issues/854) and sorry for any headaches (I'm unsure of the impact of this change, so let me know if this is a huge hassle for you). #### Install Run `dprint upgrade` or see https://dprint.dev/install/ #### Checksums |Artifact|SHA-256 Checksum| |:--|:--| |dprint-x86\_64-apple-darwin.zip|cdea84bce1d84c26e8eced2265d246b79a849ec2e7d1377d98dd7bdb21c7ce83| |dprint-aarch64-apple-darwin.zip|f3ff4faef83d14c3b4ae262e79a40d4e0fc3fa1903d0b6e9b82f0b25b00e9499| |dprint-x86\_64-pc-windows-msvc.zip|74e5ab38c744d5903862c2b5174d0fef9759b5506da775e1fb93b6a68c63101d| |dprint-x86\_64-pc-windows-msvc-installer.exe|107786c41be76b49463a50d7d9d788397bba723e107e723347f8e8dde65339dc| |dprint-x86\_64-unknown-linux-gnu.zip|cb72fa6b474e2847a3cf5705b43ee2cbfdafddd7c69ff162309fd1f4f43c872a| |dprint-x86\_64-unknown-linux-musl.zip|4a7d6fa6b920ab150f580965556086cdd7992e07078e627ab9a9d1c3bd30ba85| |dprint-aarch64-unknown-linux-gnu.zip|c4e892d5d237a57ede7900255e5ce669b56160e61c89798c118fbd4c36d48ff2| |dprint-aarch64-unknown-linux-musl.zip|e2b6d87167d21f1f01571790e79526ef9caff3b8b75f5cac348c4f06f60a8c16| ### [`v0.46.0`](https://togithub.com/dprint/dprint/releases/tag/0.46.0) [Compare Source](https://togithub.com/dprint/dprint/compare/0.45.1...0.46.0) ##### Changes - feat: gitignore support ([#832](https://togithub.com/dprint/dprint/issues/832)) - feat: `DPRINT_TLS_CA_STORE` and `DPRINT_CERT` ([#850](https://togithub.com/dprint/dprint/issues/850)) - fix: remove BOM handling from the CLI ([#844](https://togithub.com/dprint/dprint/issues/844)) Please run `dprint config update` after upgrading as some BOM handling happens in the plugins now and some may fail to parse files with a BOM now. See [#854](https://togithub.com/dprint/dprint/issues/854) and sorry for any headaches (I'm unsure of the impact of this change, so let me know if this is a huge hassle for you). ##### Install Run `dprint upgrade` or see https://dprint.dev/install/ ##### Checksums |Artifact|SHA-256 Checksum| |:--|:--| |dprint-x86\_64-apple-darwin.zip|e339f1f891c60087676d72f70ba5f80dcaedde4bdc58730b9cb68a5483b3abfd| |dprint-aarch64-apple-darwin.zip|4b608b3676f10e04328c3d8be396bded96328ebca9b95b70bf5baf67bed7b135| |dprint-x86\_64-pc-windows-msvc.zip|786201545938f6f7c6d407e6404b31ae9bbf9e5a4abc4c88dc9bd73da369a906| |dprint-x86\_64-pc-windows-msvc-installer.exe|e445b37af124e5d8ef691685632509d2bfc701962c58db89eebc8a8de7352ab4| |dprint-x86\_64-unknown-linux-gnu.zip|8274ea44d2ab4d10b8bdfcc824d946a6d051594aede49c9db8c5e810887abd67| |dprint-x86\_64-unknown-linux-musl.zip|7a2c12edc868259be890174c4ec3bd51c81ec8773aa294e12fac0634f36d15f5| |dprint-aarch64-unknown-linux-gnu.zip|6617465acba53c9b939e73f20538a8027e45593342c34f7ac4826c9f4e6cf53b| |dprint-aarch64-unknown-linux-musl.zip|e52c0a3398e34e88ffe560e719bf8361ba3f35b4e0927ab9ba0761796884ce24| </details> <details> <summary>fujiwara/awslim (fujiwara/awslim)</summary> ### [`v0.1.2`](https://togithub.com/fujiwara/awslim/releases/tag/v0.1.2) [Compare Source](https://togithub.com/fujiwara/awslim/compare/v0.1.1...v0.1.2) #### What's Changed - feature: Add suport for building with specified OS/Arch by [@ToshihitoKon](https://togithub.com/ToshihitoKon) in [fujiwara/awslim#20 - merge [#20](https://togithub.com/fujiwara/awslim/issues/20) into main by [@fujiwara](https://togithub.com/fujiwara) in [fujiwara/awslim#21 #### New Contributors - [@ToshihitoKon](https://togithub.com/ToshihitoKon) made their first contribution in [fujiwara/awslim#20 **Full Changelog**: fujiwara/awslim@v0.1.1...v0.1.2 </details> <details> <summary>golangci/golangci-lint (golangci/golangci-lint)</summary> ### [`v1.59.0`](https://togithub.com/golangci/golangci-lint/compare/v1.58.2...v1.59.0) [Compare Source](https://togithub.com/golangci/golangci-lint/compare/v1.58.2...v1.59.0) </details> <details> <summary>simulot/immich-go (simulot/immich-go)</summary> ### [`v0.15.0`](https://togithub.com/simulot/immich-go/releases/tag/0.15.0) [Compare Source](https://togithub.com/simulot/immich-go/compare/0.14.1...0.15.0) ##### fix [#255](https://togithub.com/simulot/immich-go/issues/255) Last percents of google puzzle solving are very slow when processing very large takeout archive The google puzzle solving is now much faster for large takeout archives. ##### fix [#215](https://togithub.com/simulot/immich-go/issues/215) Use XDG_CONFIG_HOME for storing config The configuration file that contains the server and the key is now stored by default in following folder: - Linux `$HOME/.config/immich-go/immich-go.json` - Windows `%AppData%\immich-go\immich-go.json` - Apple `$HOME/Library/Application Support/immich-go/immich-go.json` ##### Store the log files into sensible dir for user's system The default log file is: - Linux `$HOME/.cache/immich-go/immich-go_YYYY-MM-DD_HH-MI-SS.log` - Windows `%LocalAppData%\immich-go\immich-go_YYYY-MM-DD_HH-MI-SS.log` - Apple `$HOME/Library/Caches/immich-go/immich-go_YYYY-MM-DD_HH-MI-SS.log` ##### Feat: \[[#249](https://togithub.com/simulot/immich-go/issues/249)] Fix Display the path of log file name The log file name is printed when the program exits. #### Changelog - [`87471f6`](https://togithub.com/simulot/immich-go/commit/87471f6) Edit releases.md - [`d4269ee`](https://togithub.com/simulot/immich-go/commit/d4269ee) Fix Display the path of log file name ([#251](https://togithub.com/simulot/immich-go/issues/251)) - [`49bf673`](https://togithub.com/simulot/immich-go/commit/49bf673) Fix Display the path of log file name ([#256](https://togithub.com/simulot/immich-go/issues/256)) - [`73fde8d`](https://togithub.com/simulot/immich-go/commit/73fde8d) Fixes [#255](https://togithub.com/simulot/immich-go/issues/255) - [`9b70932`](https://togithub.com/simulot/immich-go/commit/9b70932) Merge branch 'main' into simulot/issue215 - [`734bbeb`](https://togithub.com/simulot/immich-go/commit/734bbeb) Merge pull request [#257](https://togithub.com/simulot/immich-go/issues/257) from simulot:simulot/issue255 - [`8805ca7`](https://togithub.com/simulot/immich-go/commit/8805ca7) Merge pull request [#258](https://togithub.com/simulot/immich-go/issues/258) from simulot:simulot/issue215 - [`c63dc09`](https://togithub.com/simulot/immich-go/commit/c63dc09) Merge pull request [#260](https://togithub.com/simulot/immich-go/issues/260) from simulot:simulot/issue259 - [`e4d1643`](https://togithub.com/simulot/immich-go/commit/e4d1643) Set a timeout for all http queries ([#253](https://togithub.com/simulot/immich-go/issues/253)) - [`f968bd6`](https://togithub.com/simulot/immich-go/commit/f968bd6) The log doesn't show discared files Fixes [#259](https://togithub.com/simulot/immich-go/issues/259) - [`37f4b83`](https://togithub.com/simulot/immich-go/commit/37f4b83) Use XDG_CONFIG_HOME for storing config ([#248](https://togithub.com/simulot/immich-go/issues/248)) - [`755e1b7`](https://togithub.com/simulot/immich-go/commit/755e1b7) Use XDG_CONFIG_HOME for storing config Fixes [#215](https://togithub.com/simulot/immich-go/issues/215) </details> <details> <summary>snyk/cli (snyk/cli)</summary> ### [`v1.1291.1`](https://togithub.com/snyk/cli/releases/tag/v1.1291.1) [Compare Source](https://togithub.com/snyk/cli/compare/v1.1291.0...v1.1291.1) The Snyk CLI is being deployed to different deployment channels, users can select the stability level according to their needs. For details please see [this documentation](https://docs.snyk.io/snyk-cli/releases-and-channels-for-the-snyk-cli) ##### Bug Fixes - **dependencies:** Upgrade go-getter to v1.7.4 to fix vulnerabilities ([#5252](https://togithub.com/snyk/snyk/issues/5252)) </details> <details> <summary>twpayne/chezmoi (twpayne/chezmoi)</summary> ### [`v2.48.2`](https://togithub.com/twpayne/chezmoi/compare/v2.48.1...v2.48.2) [Compare Source](https://togithub.com/twpayne/chezmoi/compare/v2.48.1...v2.48.2) </details> <details> <summary>weaveworks/eksctl (weaveworks/eksctl)</summary> ### [`v0.180.0`](https://togithub.com/eksctl-io/eksctl/releases/tag/v0.180.0): eksctl 0.180.0 [Compare Source](https://togithub.com/weaveworks/eksctl/compare/0.179.0-rc.0...0.180.0) ### Release v0.180.0 #### 🐛 Bug Fixes - Add option to create service account for pod identities which defaults to `false` ([#7784](https://togithub.com/weaveworks/eksctl/issues/7784)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 4pm on thursday" in timezone America/Los_Angeles, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/scottames/dots).  --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: scottames-github-bot[bot] <162828115+scottames-github-bot[bot]@users.noreply.github.com>

sipsma requested review from vito and jedevc May 13, 2024 18:56

sipsma added this to the next milestone May 13, 2024

sipsma force-pushed the no-more-shim branch 2 times, most recently from cda5f44 to 9a055e1 Compare May 15, 2024 20:09

sipsma modified the milestones: next, v0.11.5 May 16, 2024

sipsma force-pushed the no-more-shim branch from 7eead48 to 2bea6bd Compare May 16, 2024 20:35

sipsma marked this pull request as ready for review May 17, 2024 01:40

jedevc reviewed May 17, 2024

View reviewed changes

vito approved these changes May 17, 2024

View reviewed changes

sipsma force-pushed the no-more-shim branch from 48e774a to d73a79f Compare May 18, 2024 00:21

sipsma added 14 commits May 17, 2024 17:47

core: migrate gpu setup from shim to executor

4b4baaf

Not much of note here, just update the OCI spec from the executor rather than the shim. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

move some initial nested client setup to executor

367382c

Signed-off-by: Erik Sipsma <erik@sipsma.dev>

core: fix handling of non-CNI containers

4429aef

Buildkit frontend implementations sometimes end up creating containers with no network enabled at all, which wasn't handled well previously. Signed-off-by: Erik Sipsma <erik@sipsma.dev>

sipsma force-pushed the no-more-shim branch from d73a79f to 9e239f2 Compare May 18, 2024 01:49

sipsma merged commit e4a11c2 into dagger:main May 20, 2024
93 checks passed

sipsma mentioned this pull request May 20, 2024

WIP refactor of server/sessions/buildkit-interfaces #7315

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove shim #7367

Remove shim #7367

sipsma commented May 13, 2024 •

edited

sipsma commented May 17, 2024

jedevc left a comment

jedevc May 17, 2024

sipsma May 17, 2024

sipsma May 17, 2024

sipsma May 17, 2024 •

edited

jedevc May 20, 2024

jedevc May 17, 2024

sipsma May 17, 2024

jedevc May 20, 2024

vito left a comment

vito May 17, 2024

sipsma May 18, 2024

vito May 17, 2024

vito May 17, 2024

vito May 17, 2024

sipsma commented May 18, 2024

sipsma commented May 20, 2024

		// we don't need a full container, just a CNI provisioned network namespace to listen in
		netNS, err := bk.NewNetworkNamespace(ctx, fullHost)

Remove shim #7367

Remove shim #7367

Conversation

sipsma commented May 13, 2024 • edited

sipsma commented May 17, 2024

jedevc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sipsma May 17, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vito left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sipsma commented May 18, 2024

sipsma commented May 20, 2024

sipsma commented May 13, 2024 •

edited

sipsma May 17, 2024 •

edited