Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dotfiles cloned before ssh-agent is started #1034

Open
martinbiard opened this issue Apr 28, 2024 · 29 comments
Open

Dotfiles cloned before ssh-agent is started #1034

martinbiard opened this issue Apr 28, 2024 · 29 comments
Labels
kind/bug windows

Comments

@martinbiard
Copy link

What happened?

I tried to use the dotfiles feature. I prefer to keep my dotfiles repository private, so SSH authentication needs to work in the devcontainer before the dotfiles repository can be cloned. Here is part of debug log relating to this issue:

[19:43:50] debug Done executing ssh server helper command
[19:43:50] debug Done creating devcontainer
[19:43:50] info Run 'ssh svg-io.devpod' to ssh into the devcontainer
[19:43:50] info Dotfiles git repository git@github.com:martinbiard/dotfiles.git specified
[19:43:50] debug Cloning dotfiles into the devcontainer...
[19:43:50] debug Running command: [C:\Program Files\DevPod\devpod-cli.exe ssh --agent-forwarding=true --start-services=true --user dev --context default svg-io --log-output=raw --command /usr/local/bin/devpod agent workspace install-dotfiles --repository git@github.com:martinbiard/dotfiles.git --debug --debug]
[19:43:50] info Adding ssh keys to agent, disable via 'devpod context set-options -o SSH_ADD_PRIVATE_KEYS=false'
[19:43:50] info Error adding private keys to ssh-agent: ssh-agent is not started

What did you expect to happen instead?

I expected dotfiles repository to be successfully cloned, regardless of if the repository is public or private.

How can we reproduce the bug? (as minimally and precisely as possible)

Create a private dotfiles repository and try to set it up in devpod.

Local Environment:

  • DevPod Version: 0.5.5
  • Operating System: windows
  • ARCH of the OS: AMD64

DevPod Provider:

  • Local/remote provider: docker
@pascalbreuninger
Copy link
Member

@martinbiard Thanks for opening the issue - fair point, we'll look into it.
In the meantime, cloning your repository using https doesn't have this problem

@shanman190
Copy link
Contributor

This looks like another one that I recently fixed here: #1008.

@pascalbreuninger
Copy link
Member

@martinbiard could you try this with the new v0.5.6 release please?

@martinbiard
Copy link
Author

martinbiard commented Apr 30, 2024 •
edited

@pascalbreuninger

Actually, the v0.5.6 pre-release is crashing really early in the build, it doesn't make it to the dotfiles part. This is from an exact setup that works on v0.5.5. I tried without anything in the Customizations > Dotfiles, and same error. I had tried a v0.5.6-alpha a few days ago and I was also getting this error back then.

It seems to be trying to get a pre-build image even though i didn't enter anything:

[00:40:31] debug Try to find prebuild image devpod-fef08a060746e0915d4f24a930109077 in repositories
[00:40:31] debug Error trying to find local image vsc-content-7c3b5:devpod-fef08a060746e0915d4f24a930109077: inspect container: []

Error response from daemon: No such image: vsc-content-7c3b5:devpod-fef08a060746e0915d4f24a930109077

exit status 1

Full debug logs of the build:

[00:40:27] debug Acquire workspace lock...
[00:40:27] debug Acquired workspace lock...
[00:40:27] info Creating devcontainer...
[00:40:27] debug Inject and run command: 'C:\Program Files\DevPod\devpod-cli.exe' helper ssh-server --stdio --debug
[00:40:27] debug Attempting to create SSH client
[00:40:27] debug Execute command locally
[00:40:27] debug SSH client created
[00:40:27] debug SSH session created
[00:40:27] debug Forwarding ssh-agent using \\.\pipe\openssh-ssh-agent
[00:40:27] info Execute SSH server command: C:\Program Files\DevPod\devpod-cli.exe helper sh -c 'C:\Program Files\DevPod\devpod-cli.exe' agent workspace up --workspace-info 'H4sIAAAAAA..........................................H7TAMAAA==' --debug
[00:40:27] info Use C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io as workspace dir
[00:40:27] debug Created logger
[00:40:27] debug Received ping from agent
[00:40:27] debug Credentials server started on port 15364...
[00:40:28] debug Incoming client connection at /
[00:40:28] debug Credentials server started...
[00:40:28] debug Create content folder C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content
[00:40:28] debug Clone Repository
[00:40:29] info Cloning into 'C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content'...
[00:40:30] debug Using docker command 'docker'
[00:40:30] debug Process OCI feature
[00:40:30] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\200bcf1c35\extracted
[00:40:30] debug Process local feature
[00:40:30] debug Parse dev container feature in C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content\.devcontainer\features\post-features
[00:40:30] debug Process local feature
[00:40:30] debug Parse dev container feature in C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content\.devcontainer\features\pre-features
[00:40:30] debug Process OCI feature
[00:40:30] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\78cbb031b4\extracted
[00:40:30] debug Process OCI feature
[00:40:30] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\93b803872e\extracted
[00:40:30] debug Process OCI feature
[00:40:30] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\da62d9222a\extracted
[00:40:30] debug Process OCI feature
[00:40:30] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\ff06937bbe\extracted
[00:40:31] debug Prebuild hash from:
[00:40:31] debug Arch: amd64
[00:40:31] debug Config: {"name":"svg-io","features":{"./features/post-features":{},"./features/pre-features":{},"ghcr.io/devcontainers/features/aws-cli:1.0.7":{"version":"2.15.2"},"ghcr.io/devcontainers/features/docker-in-docker:2.10.2":{"version":"24.0.9"},"ghcr.io/devcontainers/features/github-cli:1.0.11":{"version":"2.48.0"},"ghcr.io/devcontainers/features/node:1.4.1":{"nvmVersion":"0.39.7","version":"20.12.2"},"ghcr.io/rocker-org/devcontainer-features/apt-packages:1.0.2":{"packages":"curl,jq,nano,rsync,shfmt,tig,tree"}},"overrideFeatureInstallOrder":["./features/pre-features","ghcr.io/rocker-org/devcontainer-features/apt-packages","ghcr.io/devcontainers/features/docker-in-docker","ghcr.io/devcontainers/features/aws-cli","ghcr.io/devcontainers/features/github-cli","ghcr.io/devcontainers/features/node","./features/setup-project"],"build":{"dockerfile":"Dockerfile","context":"..","options":["--ssh","default"]}}
[00:40:31] debug DockerfileContent: FROM mcr.microsoft.com/devcontainers/base:ubuntu-22.04 AS dev_container_auto_added_stage_label

ENV container=docker

# COPY --exlude=.devpod-internal ./ /workspace/
COPY ./ /workspace/

# Remove once COPY --exlude is supported
RUN rm -rf /workspace/.devpod-internal

VOLUME ["/workspace"]

WORKDIR /workspace

EXPOSE 3000/tcp
EXPOSE 9229/tcp
EXPOSE 12557/tcp
EXPOSE 13557/tcp
[00:40:31] debug ContextHash: 6dff23bcf29a42e02280459d6bcbc9bc6f7271b495ca4d3b41930444ab850b1a
[00:40:31] debug Try to find prebuild image devpod-fef08a060746e0915d4f24a930109077 in repositories
[00:40:31] debug Error trying to find local image vsc-content-7c3b5:devpod-fef08a060746e0915d4f24a930109077: inspect container: []

Error response from daemon: No such image: vsc-content-7c3b5:devpod-fef08a060746e0915d4f24a930109077

exit status 1
[00:40:31] info Build with docker buildx...
[00:40:31] debug Running docker docker: docker buildx build -f C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content\.devpod-internal\Dockerfile-with-features --load -t vsc-content-7c3b5:devpod-fef08a060746e0915d4f24a930109077 --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg BUILDKIT_INLINE_CACHE=1 --target dev_containers_target_stage --ssh default C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content
[00:40:31] info ERROR: failed to open C:\Users\mbiard\AppData\Local\Temp\auth-agent2529056593/listener.sock: open C:\Users\mbiard\AppData\Local\Temp\auth-agent2529056593/listener.sock: The file cannot be accessed by the system.
[00:40:31] info exit status 1
[00:40:31] info build image
[00:40:31] info github.com/loft-sh/devpod/pkg/driver/docker.(*dockerDriver).buildxBuild
[00:40:31] info D:/a/devpod/devpod/pkg/driver/docker/build.go:317
[00:40:31] info github.com/loft-sh/devpod/pkg/driver/docker.(*dockerDriver).BuildDevContainer
[00:40:31] info D:/a/devpod/devpod/pkg/driver/docker/build.go:86
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildImage
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/build.go:331
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildAndExtendImage
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/build.go:187
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).build
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/build.go:30
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).runSingleContainer
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/single.go:70
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).Up
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/run.go:121
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).devPodUp
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:399
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).up
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:160
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:94
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[00:40:31] info github.com/spf13/cobra.(*Command).execute
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:916
[00:40:31] info github.com/spf13/cobra.(*Command).ExecuteC
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1044
[00:40:31] info github.com/spf13/cobra.(*Command).Execute
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:968
[00:40:31] info github.com/loft-sh/devpod/cmd.Execute
[00:40:31] info D:/a/devpod/devpod/cmd/root.go:90
[00:40:31] info main.main
[00:40:31] info D:/a/devpod/devpod/main.go:8
[00:40:31] info runtime.main
[00:40:31] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[00:40:31] info runtime.goexit
[00:40:31] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[00:40:31] info buildx build
[00:40:31] info github.com/loft-sh/devpod/pkg/driver/docker.(*dockerDriver).BuildDevContainer
[00:40:31] info D:/a/devpod/devpod/pkg/driver/docker/build.go:88
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildImage
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/build.go:331
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildAndExtendImage
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/build.go:187
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).build
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/build.go:30
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).runSingleContainer
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/single.go:70
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).Up
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/run.go:121
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).devPodUp
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:399
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).up
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:160
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:94
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[00:40:31] info github.com/spf13/cobra.(*Command).execute
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:916
[00:40:31] info github.com/spf13/cobra.(*Command).ExecuteC
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1044
[00:40:31] info github.com/spf13/cobra.(*Command).Execute
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:968
[00:40:31] info github.com/loft-sh/devpod/cmd.Execute
[00:40:31] info D:/a/devpod/devpod/cmd/root.go:90
[00:40:31] info main.main
[00:40:31] info D:/a/devpod/devpod/main.go:8
[00:40:31] info runtime.main
[00:40:31] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[00:40:31] info runtime.goexit
[00:40:31] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[00:40:31] info build image
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).runSingleContainer
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/single.go:78
[00:40:31] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).Up
[00:40:31] info D:/a/devpod/devpod/pkg/devcontainer/run.go:121
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).devPodUp
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:399
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).up
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:160
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:94
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[00:40:31] info github.com/spf13/cobra.(*Command).execute
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:916
[00:40:31] info github.com/spf13/cobra.(*Command).ExecuteC
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1044
[00:40:31] info github.com/spf13/cobra.(*Command).Execute
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:968
[00:40:31] info github.com/loft-sh/devpod/cmd.Execute
[00:40:31] info D:/a/devpod/devpod/cmd/root.go:90
[00:40:31] info main.main
[00:40:31] info D:/a/devpod/devpod/main.go:8
[00:40:31] info runtime.main
[00:40:31] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[00:40:31] info runtime.goexit
[00:40:31] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[00:40:31] info devcontainer up
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:96
[00:40:31] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[00:40:31] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[00:40:31] info github.com/spf13/cobra.(*Command).execute
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:916
[00:40:31] info github.com/spf13/cobra.(*Command).ExecuteC
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1044
[00:40:31] info github.com/spf13/cobra.(*Command).Execute
[00:40:31] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:968
[00:40:31] info github.com/loft-sh/devpod/cmd.Execute
[00:40:31] info D:/a/devpod/devpod/cmd/root.go:90
[00:40:31] info main.main
[00:40:31] info D:/a/devpod/devpod/main.go:8
[00:40:31] info runtime.main
[00:40:31] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[00:40:31] info runtime.goexit
[00:40:31] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[00:40:31] info exit status 1
[00:40:31] debug Connection to SSH Server closed
[00:40:31] debug Done creating devcontainer
[00:40:31] debug Done executing ssh server helper command
[00:40:31] fatal Process exited with status 1
run agent command
github.com/loft-sh/devpod/pkg/devcontainer/sshtunnel.ExecuteCommand.func2
        D:/a/devpod/devpod/pkg/devcontainer/sshtunnel/sshtunnel.go:126
runtime.goexit
        C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650

@pascalbreuninger
Copy link
Member

@martinbiard I see, thanks. Is C:\Users\mbiard\AppData\Local\Temp\ a readonly directory?

@martinbiard
Copy link
Author

@pascalbreuninger No, it's writeable.

@shanman190
Copy link
Contributor

So I think the issue is related to the slashes used in the path. Golang more often than not required os-specific usage. So I suspect initially that the /listener.sock is the true culprit of not being able to create the file.

@pascalbreuninger
Copy link
Member

@shanman190 Oh I missed that! Good find

@pascalbreuninger
Copy link
Member

#1042 might fix this

@pascalbreuninger
Copy link
Member

@martinbiard Can you try again with this prelease please?

@martinbiard
Copy link
Author

@pascalbreuninger Still same issue, unfortunately.

Version 0.5.7-10

Logs:

[00:32:28] debug Acquire workspace lock...
[00:32:28] debug Acquired workspace lock...
[00:32:28] info Creating devcontainer...
[00:32:28] debug Attempting to create SSH client
[00:32:28] debug Inject and run command: 'C:\Program Files\DevPod\devpod-cli.exe' helper ssh-server --stdio --debug
[00:32:28] debug Execute command locally
[00:32:28] debug SSH client created
[00:32:28] debug SSH session created
[00:32:28] debug Forwarding ssh-agent using \\.\pipe\openssh-ssh-agent
[00:32:28] info Execute SSH server command: C:\Program Files\DevPod\devpod-cli.exe helper sh -c 'C:\Program Files\DevPod\devpod-cli.exe' agent workspace up --workspace-info 'H4sIAAAAAAAA/7SSb2/TMBDGvwq....AA///C/g6J3AMAAA==' --debug
[00:32:28] info Use C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io as workspace dir
[00:32:28] debug Created logger
[00:32:28] debug Received ping from agent
[00:32:28] debug Credentials server started on port 14374...
[00:32:29] debug Incoming client connection at /
[00:32:29] debug Credentials server started...
[00:32:29] debug Create content folder C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content
[00:32:29] debug Clone Repository
[00:32:29] info Cloning into 'C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content'...
[00:32:30] debug Using docker command 'docker'
[00:32:31] debug Process OCI feature
[00:32:31] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\da62d9222a\extracted
[00:32:31] debug Process OCI feature
[00:32:31] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\ff06937bbe\extracted
[00:32:31] debug Process OCI feature
[00:32:31] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\200bcf1c35\extracted
[00:32:31] debug Process local feature
[00:32:31] debug Parse dev container feature in C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content\.devcontainer\features\post-features
[00:32:31] debug Process local feature
[00:32:31] debug Parse dev container feature in C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content\.devcontainer\features\pre-features
[00:32:31] debug Process OCI feature
[00:32:31] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\78cbb031b4\extracted
[00:32:31] debug Process OCI feature
[00:32:31] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\93b803872e\extracted
[00:32:32] debug Prebuild hash from:
[00:32:32] debug Arch: amd64
[00:32:32] debug Config: {"name":"svg-io","features":{"./features/post-features":{},"./features/pre-features":{},"ghcr.io/devcontainers/features/aws-cli:1.0.7":{"version":"2.15.2"},"ghcr.io/devcontainers/features/docker-in-docker:2.10.2":{"version":"24.0.9"},"ghcr.io/devcontainers/features/github-cli:1.0.11":{"version":"2.48.0"},"ghcr.io/devcontainers/features/node:1.4.1":{"nvmVersion":"0.39.7","version":"20.12.2"},"ghcr.io/rocker-org/devcontainer-features/apt-packages:1.0.2":{"packages":"curl,jq,nano,rsync,shfmt,tig,tree"}},"overrideFeatureInstallOrder":["./features/pre-features","ghcr.io/rocker-org/devcontainer-features/apt-packages","ghcr.io/devcontainers/features/docker-in-docker","ghcr.io/devcontainers/features/aws-cli","ghcr.io/devcontainers/features/github-cli","ghcr.io/devcontainers/features/node","./features/setup-project"],"build":{"dockerfile":"Dockerfile","context":"..","options":["--ssh","default"]}}
[00:32:32] debug DockerfileContent: FROM mcr.microsoft.com/devcontainers/base:ubuntu-22.04 AS dev_container_auto_added_stage_label

ENV container=docker

# COPY --exlude=.devpod-internal ./ /workspace/
COPY ./ /workspace/

# Remove once COPY --exlude is supported
RUN rm -rf /workspace/.devpod-internal

VOLUME ["/workspace"]

WORKDIR /workspace

EXPOSE 3000/tcp
EXPOSE 9229/tcp
EXPOSE 12557/tcp
EXPOSE 13557/tcp
[00:32:32] debug ContextHash: 0430d81f48b5bc80e0196c4eb51a7464c7e5eeb6daea0041a07bf1a559c12e96
[00:32:32] debug Try to find prebuild image devpod-fb4741cf9d8c89bede784843a0b284a5 in repositories
[00:32:32] debug Error trying to find local image vsc-content-7c3b5:devpod-fb4741cf9d8c89bede784843a0b284a5: inspect container: []

Error response from daemon: No such image: vsc-content-7c3b5:devpod-fb4741cf9d8c89bede784843a0b284a5

exit status 1
[00:32:32] info Build with docker buildx...
[00:32:32] debug Running docker docker: docker buildx build -f C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content\.devpod-internal\Dockerfile-with-features --load -t vsc-content-7c3b5:devpod-fb4741cf9d8c89bede784843a0b284a5 --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --target dev_containers_target_stage --ssh default C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svg-io\content
[00:32:32] info ERROR: failed to open C:\Users\mbiard\AppData\Local\Temp\auth-agent271879107\listener.sock: open C:\Users\mbiard\AppData\Local\Temp\auth-agent271879107\listener.sock: The file cannot be accessed by the system.
[00:32:32] info exit status 1
[00:32:32] info build image
[00:32:32] info github.com/loft-sh/devpod/pkg/driver/docker.(*dockerDriver).buildxBuild
[00:32:32] info D:/a/devpod/devpod/pkg/driver/docker/build.go:317
[00:32:32] info github.com/loft-sh/devpod/pkg/driver/docker.(*dockerDriver).BuildDevContainer
[00:32:32] info D:/a/devpod/devpod/pkg/driver/docker/build.go:86
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildImage
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/build.go:331
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildAndExtendImage
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/build.go:187
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).build
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/build.go:30
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).runSingleContainer
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/single.go:70
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).Up
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/run.go:121
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).devPodUp
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:399
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).up
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:160
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:94
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[00:32:32] info github.com/spf13/cobra.(*Command).execute
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:916
[00:32:32] info github.com/spf13/cobra.(*Command).ExecuteC
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1044
[00:32:32] info github.com/spf13/cobra.(*Command).Execute
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:968
[00:32:32] info github.com/loft-sh/devpod/cmd.Execute
[00:32:32] info D:/a/devpod/devpod/cmd/root.go:90
[00:32:32] info main.main
[00:32:32] info D:/a/devpod/devpod/main.go:8
[00:32:32] info runtime.main
[00:32:32] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[00:32:32] info runtime.goexit
[00:32:32] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[00:32:32] info buildx build
[00:32:32] info github.com/loft-sh/devpod/pkg/driver/docker.(*dockerDriver).BuildDevContainer
[00:32:32] info D:/a/devpod/devpod/pkg/driver/docker/build.go:88
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildImage
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/build.go:331
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildAndExtendImage
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/build.go:187
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).build
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/build.go:30
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).runSingleContainer
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/single.go:70
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).Up
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/run.go:121
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).devPodUp
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:399
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).up
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:160
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:94
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[00:32:32] info github.com/spf13/cobra.(*Command).execute
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:916
[00:32:32] info github.com/spf13/cobra.(*Command).ExecuteC
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1044
[00:32:32] info github.com/spf13/cobra.(*Command).Execute
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:968
[00:32:32] info github.com/loft-sh/devpod/cmd.Execute
[00:32:32] info D:/a/devpod/devpod/cmd/root.go:90
[00:32:32] info main.main
[00:32:32] info D:/a/devpod/devpod/main.go:8
[00:32:32] info runtime.main
[00:32:32] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[00:32:32] info runtime.goexit
[00:32:32] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[00:32:32] info build image
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).runSingleContainer
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/single.go:78
[00:32:32] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).Up
[00:32:32] info D:/a/devpod/devpod/pkg/devcontainer/run.go:121
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).devPodUp
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:399
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).up
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:160
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:94
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[00:32:32] info github.com/spf13/cobra.(*Command).execute
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:916
[00:32:32] info github.com/spf13/cobra.(*Command).ExecuteC
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1044
[00:32:32] info github.com/spf13/cobra.(*Command).Execute
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:968
[00:32:32] info github.com/loft-sh/devpod/cmd.Execute
[00:32:32] info D:/a/devpod/devpod/cmd/root.go:90
[00:32:32] info main.main
[00:32:32] info D:/a/devpod/devpod/main.go:8
[00:32:32] info runtime.main
[00:32:32] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[00:32:32] info runtime.goexit
[00:32:32] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[00:32:32] info devcontainer up
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:96
[00:32:32] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[00:32:32] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[00:32:32] info github.com/spf13/cobra.(*Command).execute
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:916
[00:32:32] info github.com/spf13/cobra.(*Command).ExecuteC
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1044
[00:32:32] info github.com/spf13/cobra.(*Command).Execute
[00:32:32] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:968
[00:32:32] info github.com/loft-sh/devpod/cmd.Execute
[00:32:32] info D:/a/devpod/devpod/cmd/root.go:90
[00:32:32] info main.main
[00:32:32] info D:/a/devpod/devpod/main.go:8
[00:32:32] info runtime.main
[00:32:32] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[00:32:32] info runtime.goexit
[00:32:32] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[00:32:32] info exit status 1
[00:32:32] debug Connection to SSH Server closed
[00:32:32] debug Done creating devcontainer
[00:32:32] debug Done executing ssh server helper command
[00:32:32] fatal Process exited with status 1
run agent command
github.com/loft-sh/devpod/pkg/devcontainer/sshtunnel.ExecuteCommand.func2
        D:/a/devpod/devpod/pkg/devcontainer/sshtunnel/sshtunnel.go:126
runtime.goexit
        C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650

@siccous
Copy link

siccous commented May 6, 2024

Encountering the same issue too, it prevents using newer versions (0.5.5 is latest working) on Windows.

It seems to be related to this PR #1008

@shanman190
Copy link
Contributor

@siccous, are you also expecting to be using the OpenSSH for Windows distribution (ie. The one that ships with Windows itself or a newer version via PowerShell/Win32-OpenSSH)?

@martinbiard and @siccous, what version of SSH are you folks using? ssh -V

@shanman190
Copy link
Contributor

shanman190 commented May 6, 2024 •
edited

I've tried this test using the loft-sh/devpod repo and my personal dotfiles on...

OS: Windows 10 Home
SSH: OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2
DevPod: 0.5.6 (win32, x86_64)
DevPod Provider: Docker

For my test both the main devcontainer repo cloned successfully, my dotfiles cloned successfully, and my dotfiles failed to install due to interactive prompts as I expected for this environment.

@shanman190
Copy link
Contributor

This also works on my Windows 10 Enterprise machine, but my docker desktop is broken there, so instead I'm using the AWS provider.

@martinbiard
Copy link
Author

OpenSSH version:
OpenSSH_for_Windows_8.9p1, LibreSSL 3.4.3

I think this issue went from being about dotfiles to being completely broken on Windows from v0.5.6 so I can't actually test the new dotfiles features. Personally, I'm still stuck on v0.5.5 and unable to upgrade due to above issue.

@shanman190
Copy link
Contributor

@martinbiard, would you be willing to upgrade your OpenSSH installation by following the guide here?
https://github.com/PowerShell/Win32-OpenSSH/wiki/Install-Win32-OpenSSH

What I'm pretty sure is the issue is that the source (OpenSSH 8.9) and the target (likely 9.x) are incompatible from an ssh-agent forwarding standpoint.

@martinbiard
Copy link
Author

@shanman190 I actually had the latest v9.x before and downgraded it to v8.9, last of 8.x because all the v9.x have a note saying "This is a beta-release (non-production ready)" while the v8.x don't have this note. I'll give it a try on 9.5 though.

@martinbiard
Copy link
Author

martinbiard commented May 7, 2024 •
edited

@shanman190 So yeah, still same issue:

Versions:
OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2
Docker Desktop for Windows v4.30.0
DevPod Version 0.5.7-alpha.2

Docker provider

Logs:

[13:11:52] debug Acquire workspace lock...
[13:11:52] debug Acquired workspace lock...
[13:11:52] info Creating devcontainer...
[13:11:52] debug Inject and run command: 'C:\Program Files\DevPod\devpod-cli.exe' helper ssh-server --stdio --debug
[13:11:52] debug Attempting to create SSH client
[13:11:52] debug Execute command locally
[13:11:52] debug SSH client created
[13:11:52] debug SSH session created
[13:11:52] debug Forwarding ssh-agent using \\.\pipe\openssh-ssh-agent
[13:11:52] info Execute SSH server command: C:\Program Files\DevPod\devpod-cli.exe helper sh -c 'C:\Program Files\DevPod\devpod-cli.exe' agent workspace up --workspace-info 'H4sIAAAAAAAA/7RSUU/bTBD8K5/2+W...AP//RLeX1AAEAAA=' --debug
[13:11:52] info Use C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svgio as workspace dir
[13:11:52] debug Created logger
[13:11:52] debug Received ping from agent
[13:11:52] debug Credentials server started on port 13475...
[13:11:53] debug Incoming client connection at /
[13:11:53] debug Credentials server started...
[13:11:53] debug Create content folder C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svgio\content
[13:11:53] debug Clone Repository
[13:11:53] info Cloning into 'C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svgio\content'...
[13:11:55] debug Using docker command 'docker'
[13:11:55] debug Process OCI feature
[13:11:55] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\ff06937bbe\extracted
[13:11:55] debug Process OCI feature
[13:11:55] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\200bcf1c35\extracted
[13:11:55] debug Process local feature
[13:11:55] debug Parse dev container feature in C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svgio\content\.devcontainer\features\post-features
[13:11:55] debug Process local feature
[13:11:55] debug Parse dev container feature in C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svgio\content\.devcontainer\features\pre-features
[13:11:55] debug Process OCI feature
[13:11:55] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\78cbb031b4\extracted
[13:11:55] debug Process OCI feature
[13:11:55] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\93b803872e\extracted
[13:11:55] debug Process OCI feature
[13:11:55] debug Parse dev container feature in C:\Users\mbiard\AppData\Local\Temp\devpod\features\da62d9222a\extracted
[13:11:56] debug Prebuild hash from:
[13:11:56] debug Arch: amd64
[13:11:56] debug Config: {"name":"svgio","features":{"./features/post-features":{},"./features/pre-features":{},"ghcr.io/devcontainers/features/aws-cli:1.0.7":{"version":"2.15.2"},"ghcr.io/devcontainers/features/docker-in-docker:2.10.2":{"version":"24.0.9"},"ghcr.io/devcontainers/features/github-cli:1.0.11":{"version":"2.48.0"},"ghcr.io/devcontainers/features/node:1.4.1":{"nvmVersion":"0.39.7","version":"20.12.2"},"ghcr.io/rocker-org/devcontainer-features/apt-packages:1.0.2":{"packages":"curl,jq,nano,postgresql-client-16,rsync,shfmt,tig,tree,libgl1,libglib2.0-0"}},"overrideFeatureInstallOrder":["./features/pre-features","ghcr.io/rocker-org/devcontainer-features/apt-packages","ghcr.io/devcontainers/features/docker-in-docker","ghcr.io/devcontainers/features/aws-cli","ghcr.io/devcontainers/features/github-cli","ghcr.io/devcontainers/features/node","./features/post-features"],"build":{"dockerfile":"Dockerfile","context":"..","options":["--ssh","default"]}}
[13:11:56] debug DockerfileContent: FROM mcr.microsoft.com/devcontainers/base:ubuntu-22.04 AS dev_container_auto_added_stage_label

ENV container=docker

# COPY --exlude=.devpod-internal ./ /workspace/
COPY ./ /workspace/

# Remove once COPY --exlude is supported
RUN rm -rf /workspace/.devpod-internal

VOLUME ["/workspace"]

WORKDIR /workspace

EXPOSE 3000/tcp
EXPOSE 9229/tcp
EXPOSE 12557/tcp
EXPOSE 13557/tcp
[13:11:56] debug ContextHash: ba1407c224fd8bae6c0534d6984d4a91da27d5f15d80e8918d210e4f8eef2be8
[13:11:56] debug Try to find prebuild image devpod-ac8005b3a5d139253b68344faef18158 in repositories
[13:11:56] debug Error trying to find local image vsc-content-5752e:devpod-ac8005b3a5d139253b68344faef18158: inspect container: []

Error response from daemon: No such image: vsc-content-5752e:devpod-ac8005b3a5d139253b68344faef18158

exit status 1
[13:11:56] info Build with docker buildx...
[13:11:56] debug Running docker docker: docker buildx build -f C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svgio\content\.devpod-internal\Dockerfile-with-features --load -t vsc-content-5752e:devpod-ac8005b3a5d139253b68344faef18158 --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label --build-arg _DEV_CONTAINERS_IMAGE_USER=root --target dev_containers_target_stage --ssh default C:\Users\mbiard\.devpod\agent\contexts\default\workspaces\svgio\content
[13:11:57] info ERROR: failed to open C:\Users\mbiard\AppData\Local\Temp\auth-agent2098067750\listener.sock: open C:\Users\mbiard\AppData\Local\Temp\auth-agent2098067750\listener.sock: The file cannot be accessed by the system.
[13:11:57] info exit status 1
[13:11:57] info build image
[13:11:57] info github.com/loft-sh/devpod/pkg/driver/docker.(*dockerDriver).buildxBuild
[13:11:57] info D:/a/devpod/devpod/pkg/driver/docker/build.go:317
[13:11:57] info github.com/loft-sh/devpod/pkg/driver/docker.(*dockerDriver).BuildDevContainer
[13:11:57] info D:/a/devpod/devpod/pkg/driver/docker/build.go:86
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildImage
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/build.go:331
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildAndExtendImage
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/build.go:187
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).build
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/build.go:30
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).runSingleContainer
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/single.go:70
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).Up
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/run.go:121
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).devPodUp
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:399
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).up
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:160
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:94
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[13:11:57] info github.com/spf13/cobra.(*Command).execute
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:940
[13:11:57] info github.com/spf13/cobra.(*Command).ExecuteC
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1068
[13:11:57] info github.com/spf13/cobra.(*Command).Execute
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:992
[13:11:57] info github.com/loft-sh/devpod/cmd.Execute
[13:11:57] info D:/a/devpod/devpod/cmd/root.go:90
[13:11:57] info main.main
[13:11:57] info D:/a/devpod/devpod/main.go:8
[13:11:57] info runtime.main
[13:11:57] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[13:11:57] info runtime.goexit
[13:11:57] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[13:11:57] info buildx build
[13:11:57] info github.com/loft-sh/devpod/pkg/driver/docker.(*dockerDriver).BuildDevContainer
[13:11:57] info D:/a/devpod/devpod/pkg/driver/docker/build.go:88
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildImage
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/build.go:331
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).buildAndExtendImage
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/build.go:187
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).build
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/build.go:30
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).runSingleContainer
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/single.go:70
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).Up
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/run.go:121
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).devPodUp
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:399
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).up
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:160
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:94
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[13:11:57] info github.com/spf13/cobra.(*Command).execute
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:940
[13:11:57] info github.com/spf13/cobra.(*Command).ExecuteC
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1068
[13:11:57] info github.com/spf13/cobra.(*Command).Execute
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:992
[13:11:57] info github.com/loft-sh/devpod/cmd.Execute
[13:11:57] info D:/a/devpod/devpod/cmd/root.go:90
[13:11:57] info main.main
[13:11:57] info D:/a/devpod/devpod/main.go:8
[13:11:57] info runtime.main
[13:11:57] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[13:11:57] info runtime.goexit
[13:11:57] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[13:11:57] info build image
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).runSingleContainer
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/single.go:78
[13:11:57] info github.com/loft-sh/devpod/pkg/devcontainer.(*runner).Up
[13:11:57] info D:/a/devpod/devpod/pkg/devcontainer/run.go:121
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).devPodUp
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:399
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).up
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:160
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:94
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[13:11:57] info github.com/spf13/cobra.(*Command).execute
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:940
[13:11:57] info github.com/spf13/cobra.(*Command).ExecuteC
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1068
[13:11:57] info github.com/spf13/cobra.(*Command).Execute
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:992
[13:11:57] info github.com/loft-sh/devpod/cmd.Execute
[13:11:57] info D:/a/devpod/devpod/cmd/root.go:90
[13:11:57] info main.main
[13:11:57] info D:/a/devpod/devpod/main.go:8
[13:11:57] info runtime.main
[13:11:57] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[13:11:57] info runtime.goexit
[13:11:57] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[13:11:57] info devcontainer up
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.(*UpCmd).Run
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:96
[13:11:57] info github.com/loft-sh/devpod/cmd/agent/workspace.NewUpCmd.func1
[13:11:57] info D:/a/devpod/devpod/cmd/agent/workspace/up.go:52
[13:11:57] info github.com/spf13/cobra.(*Command).execute
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:940
[13:11:57] info github.com/spf13/cobra.(*Command).ExecuteC
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:1068
[13:11:57] info github.com/spf13/cobra.(*Command).Execute
[13:11:57] info D:/a/devpod/devpod/vendor/github.com/spf13/cobra/command.go:992
[13:11:57] info github.com/loft-sh/devpod/cmd.Execute
[13:11:57] info D:/a/devpod/devpod/cmd/root.go:90
[13:11:57] info main.main
[13:11:57] info D:/a/devpod/devpod/main.go:8
[13:11:57] info runtime.main
[13:11:57] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/proc.go:267
[13:11:57] info runtime.goexit
[13:11:57] info C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650
[13:11:57] info exit status 1
[13:11:57] debug Connection to SSH Server closed
[13:11:57] debug Done creating devcontainer
[13:11:57] debug Done executing ssh server helper command
[13:11:57] fatal Process exited with status 1
run agent command
github.com/loft-sh/devpod/pkg/devcontainer/sshtunnel.ExecuteCommand.func2
        D:/a/devpod/devpod/pkg/devcontainer/sshtunnel/sshtunnel.go:126
runtime.goexit
        C:/hostedtoolcache/windows/go/1.21.8/x64/src/runtime/asm_amd64.s:1650

@siccous
Copy link

siccous commented May 7, 2024

@shanman190 and @martinbiard

Found the issue, it's caused by ssh mount https://docs.docker.com/build/building/secrets/#ssh-mounts

Both me and Martin are using this feature in our build, if you remove it then it works.

@shanman190 you should be able to reproduce it now.

@martinbiard
Copy link
Author

@shanman190 Ok I did some more testing and found that, if I use a devcontainer image directly in the devcontainer.json it does work, but in my case I'm actually using the build my own image that extends from one of the official devcontainer images.

So, this config works:

{
    "name": "Node.js",
    "image": "mcr.microsoft.com/devcontainers/javascript-node:0-18-bullseye",
    "customizations": {
      "vscode": {
        "settings": {},
        "extensions": [
          "streetsidesoftware.code-spell-checker"
        ]
      }
    },
    "portsAttributes": {
      "3000": {
        "label": "Hello Remote World",
        "onAutoForward": "notify"
      }
    },
  }

While this one doesn't:

{
  "name": "svgio",
  "build": {
    "dockerfile": "Dockerfile",
    "context": "..",
    "options": ["--ssh", "default"]
  },
  "runArgs": ["--name", "svgio", "--hostname", "svgio", "--gpus", "all"],
  "forwardPorts": [3000, 9229, 12557, 13557],
  "portsAttributes": {
    "3000": {
      "label": "Website",
      "onAutoForward": "notify",
      "requireLocalPort": true
    },
    "9229": {
      "label": "Debugger",
      "onAutoForward": "notify",
      "requireLocalPort": true
    },
    "12557": {
      "label": "SST",
      "onAutoForward": "notify",
      "requireLocalPort": true
    },
    "13557": {
      "label": "SST",
      "onAutoForward": "notify",
      "requireLocalPort": true
    }
  },
  "workspaceMount": "type=volume,source=svgio,target=/workspace",
  "workspaceFolder": "/workspace",
  "remoteUser": "dev"
}

Dockerfile:

FROM mcr.microsoft.com/devcontainers/base:ubuntu-22.04

ENV container=docker

COPY ./ /workspace/
RUN rm -rf /workspace/.devpod-internal

VOLUME ["/workspace"]

WORKDIR /workspace

EXPOSE 3000/tcp
EXPOSE 9229/tcp
EXPOSE 12557/tcp
EXPOSE 13557/tcp

@martinbiard
Copy link
Author

@shanman190 @siccous

I can confirm, simply removing this from my config allows the build to proceed without error:
"options": ["--ssh", "default"]

@shanman190
Copy link
Contributor

Just to check, what docker distribution and version are you all using from a local provider? I just want to make sure that I'm covering my base on that side as well.

@siccous
Copy link

siccous commented May 7, 2024

@shanman190

Docker version: 26.0.0, build 2ae903e (WSL2 integration through Docker Desktop)

@martinbiard
Copy link
Author

@shanman190

Versions:
OpenSSH_for_Windows_9.5p1, LibreSSL 3.8.2
Docker version 26.1.1, build 4cf5afa
Docker Desktop for Windows v4.30.0
DevPod Version 0.5.7-alpha.2

@shanman190
Copy link
Contributor

shanman190 commented May 12, 2024 •
edited

@martinbiard, so the option --ssh is instructing the Docker build container with agent forward during the container build itself. Based upon your Dockerfile, you're not using the SSH agent during the build, so you could just outright remove those options.

@siccous, are you performing a Git SSH clone within your Dockerfile and thus would be actually using the SSH forwarding into the Docker build?

As to what appears to be the root cause of the issues is that the ssh-server helper for the docker path is started on the source machine expecting a unix like environment. The ssh server code on the other hand doesn't know how to handle the proxying to the Windows pipe, unlike the core devpod process. Since the docker build is happening from within the agent process tree, this means that it inherits the SSH_AUTH_SOCK environment variable which points to the listener socket that the SSH server establishes. Due to the Windows pipe though, this socket ends up at a dead end. Docker buildx has knowledge about how to natively use the Windows socket which you can enable with the --ssh option (though again in Martin's case it's not needed). Like devpod, Docker prefers the Unix socket, then falls back to the Windows named pipe.

As I see it, there are probably two options to actually solve the issue of allowing the SSH agent within the Docker build itself:

  • In the devpod flow for the Docker provider, don't start the SSH server or set the SSH_AUTH_SOCK which will result in the local Docker process automatically using the Windows socket.
  • Update the SSH server code to be OS aware and connect to the Windows named pipe directly. (Given there is no sshtunnel being used, this is why that code path isn't working right here)

@pascalbreuninger, any opinions here?

@pascalbreuninger
Copy link
Member

pascalbreuninger commented May 12, 2024 •
edited

@shanman190 I see, thanks for investigating. I didn't really follow the discussion earlier and wasn't aware the build arg --ssh is causing all of these issues, good to know.

As for the solutions

In the devpod flow for the Docker provider, don't start the SSH server or set the SSH_AUTH_SOCK which will result in the local Docker process automatically using the Windows socket.

This relies on docker actually handling the windows forwarding and as drop-in-replacements (podman, orbstack, colima) become more common I wouldn't rely on all of these third party binaries to handle it correctly, so

Update the SSH server code to be OS aware and connect to the Windows named pipe directly. (Given there is no sshtunnel being used, this is why that code path isn't working right here)

Seems to be the way to go

@martinbiard
Copy link
Author

martinbiard commented May 13, 2024 •
edited

@shanman190 You are right, in my case it's not needed, it was a remnant. Previously, I tried Jetbrains Gateway directly without DevPod and Gateway was doing a shallow git checkout so inside my container I didn't have a properly setup git repository to work with and I was force to re-clone inside the devcontainer which is why I was needing the SSH agent forwarding at build-time.

@siccous
Copy link

siccous commented May 14, 2024

@shanman190

Yes in my case it is needed, we are cloning some internal repos through SSH during the build.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@89luca89 @shanman190 @martinbiard @siccous @pascalbreuninger