Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Remote-SSH Bug]: Server installation corrupted, won't recover #9823

Closed
3 tasks done
JamyDev opened this issue Apr 24, 2024 · 1 comment · Fixed by microsoft/vscode#211500
Closed
3 tasks done

[Remote-SSH Bug]: Server installation corrupted, won't recover #9823

JamyDev opened this issue Apr 24, 2024 · 1 comment · Fixed by microsoft/vscode#211500
Assignees
@connor4312
Labels
bug Issue identified by VS Code Team member as probable bug ssh Issue in vscode-remote SSH
Milestone
May 2024

Comments

@JamyDev
Copy link

JamyDev commented Apr 24, 2024 •
edited

Is there an existing issue for this bug?

  • I have searched the existing issues

Required Troubleshooting Steps

  • I have followed these troubleshooting steps
  • I have tried both values of the remote.SSH.useLocalServer setting

Connect Locally

It connects successfully

->

useLocalServer: true -> fails with file not found error

[15:58:22.613] > ready: 39ca82d963d2
[15:58:22.654] > Linux 5.10.197+ #1 SMP Fri Dec 29 09:13:24 UTC 2023
[15:58:22.655] Platform: linux
[15:58:22.693] > /bin/zsh
[15:58:22.694] Shell: zsh
[15:58:22.694] Creating bash subshell inside "zsh"
[15:58:22.804] > bash version:  5.0.3(1)-release
[15:58:22.841] > bash version:  5.0.3(1)-release
> bash version:  5.0.3(1)-release
[15:58:22.861] > 39ca82d963d2: running
> Found existing installation at /home/user/.vscode-server...
> Starting VS Code CLI... "/home/user/.vscode-server/code-e170252f762678dec6ca2cc69aba1570769a5d39" command-shell --cli-data-dir "/home/user/.vscode-server/cli" --on-port --parent-process-id 391372 &> "/home/user/.vscode-server/.cli.e170252f762678dec6ca2cc69aba1570769a5d39.log" < /dev/null
> printenv:
[15:58:22.877] >     SHELL=/bin/zsh
>     DOCKER_TLS_VERIFY=1
>     NVM_INC=/opt/nodejs/.nvm/versions/node/v16.15.0/include/node
>     JAVA_HOME=/usr/lib/jvm/temurin-8-jdk-amd64
>     SSH_AUTH_SOCK=/var/lib/remotepod/ssh/active_ssh_auth_sock
>     REMOTEPOD_USER_FULL_NAME=
>     REMOTEPOD_NAME=user
>     PWD=/home/user
>     KUBERNETES_REMOTEPOD_HOST_IP=10.138.0.41
>     LOGNAME=user
>     UBAZEL_PROFILE_ACTIONS=critical_path
>     REMOTEPOD_SHELL=/bin/zsh
>     REMOTEPOD_IDE_CONNECTED_CLIENTS=/opt/gateway/connected_clients
>     ORG_JETBRAINS_PROJECTOR_SERVER_ENABLE_WS_TRANSPORT=true
>     KUBERNETES_REMOTEPOD_CPU_LIMIT=48
>     HOME=/home/user
>     REMOTEPOD_ENVIRONMENT=kubernetes
>     CWM_TRUST_PROJECTS=1
>     LDA_HOME=/opt/lda
>     TMPDIR=/tmp
>     VSCODE_AGENT_FOLDER=/home/user/.vscode-server
>     SSH_CONNECTION=127.16.150.9 34268 127.17.145.32 22
>     REMOTEPOD_TELEGRAF_PORT_HTTP=49090
>     REMOTEPOD_RELEASE_CHANNEL=stable
>     NVM_DIR=/opt/nodejs/.nvm
>     REMOTEPOD_DISK_SIZE=256Gi
>     KUBERNETES_REMOTEPOD_MEM_LIMIT=96
>     UBAZEL_PROFILE_SAMPLING=100
>     KUBERNETES_REMOTEPOD_CPU_REQUEST=2
>     USER=user
>     DOCKER_CERT_PATH=/home/user/.docker
>     NPM_CONFIG_GLOBALCONFIG=/etc/npmrc
>     SHLVL=1
>     NVM_CD_FLAGS=-q
>     REMOTEPOD_METRICS_PATH=/var/lib/remotepod/metrics
>     KUBERNETES_REMOTEPOD_POD_IP=127.17.145.32
>     KUBERNETES_REMOTEPOD_POD_NAME=user-865744db97-tbwv5
>     KUBERNETES_REMOTEPOD_MEM_REQUEST=16
>     GO111MODULE=on
>     SSH_CLIENT=10.16.150.9 34268 22
>     LC_ALL=en_US.UTF-8
>     PATH=/opt/nodejs/.nvm/versions/node/v16.15.0/bin:/opt/lda/pex_resources/scripts/binaries:/home/user/bin:/opt/git/bin:/home/user/go-repos:/opt/go/path/bin:/opt/go/root/bin:/usr/lib/jvm/temurin-8-jdk-amd64/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/nodejs/node_modules/bin
>     KUBERNETES_REMOTEPOD_NODE_NAME=gke-remotepod-remotepod-1-24-highmem-af79f649-7vws
>     REMOTEPOD_VERSION=0.4.58
>     REMOTEPOD_FLAVOR=go
>     NVM_BIN=/opt/nodejs/.nvm/versions/node/v16.15.0/bin
>     MAIL=/var/mail/user
>     OLDPWD=/home/user
>     GOPATH=/home/user/go-repos:/opt/go/path
>     _=/usr/bin/printenv
> Removing old logfile at /home/user/.vscode-server/.cli.e170252f762678dec6ca2cc69aba1570769a5d39.log
> Spawned remote CLI: 391392
> Waiting for server log...
[15:58:22.912] > 39ca82d963d2: start
> DISPLAY====
> listeningOn==127.0.0.1:40253==
> osReleaseId==debian==
> arch==x86_64==
> vscodeArch==x64==
> bitness==64==
> tmpDir==/tmp==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
> serverStartTime==45==
> execServerToken==a1111aa1-1a1a-1a1a-1aa1-1111a111aa1a==
> 39ca82d963d2: end
[15:58:22.913] Received install output:
DISPLAY====
listeningOn==127.0.0.1:40253==
osReleaseId==debian==
arch==x86_64==
vscodeArch==x64==
bitness==64==
tmpDir==/tmp==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
serverStartTime==45==
execServerToken==a1111aa1-1a1a-1a1a-1aa1-1111a111aa1a==

[15:58:22.913] Remote server is listening on port 40253
[15:58:22.913] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":40253},"osReleaseId":"debian","arch":"x86_64","display":"","tmpDir":"/tmp","platform":"linux","execServerToken":"a1111aa1-1a1a-1a1a-1aa1-1111a111aa1a"},"serverStartTime":45,"installUnpackCode":""}
[15:58:22.914] Persisting server connection details to /Users/user/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-57434044-e170252f762678dec6ca2cc69aba1570769a5d39-0.110.1-es/data.json
[15:58:22.919] Starting forwarding server. local port 63895 -> socksPort 63878 -> remotePort 40253
[15:58:22.920] Forwarding server listening on port 63895
[15:58:22.920] Waiting for ssh tunnel to be ready
[15:58:22.921] [Forwarding server port 63895] Got connection 0
[15:58:22.922] Tunneled port 40253 to local port 63895
[15:58:22.922] Resolved "ssh-remote+remote-server.internal" to "port 63895"
[15:58:22.981] Verified and reusing cached exec server for ssh-remote+remote-server.internal
[15:58:22.982] ------




[15:58:23.024] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.024] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.024] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.024] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.027] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.028] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.028] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.028] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.029] [server] Server setup complete
[15:58:23.029] [server] Server setup complete
[15:58:23.029] [server] Server setup complete
[15:58:23.029] [server] Server setup complete
[15:58:23.030] [server] Starting server...
[15:58:23.030] [server] Starting server...
[15:58:23.030] [server] Starting server...
[15:58:23.030] [server] Starting server...
[15:58:23.030] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-70f6988a-3420-43c2-b0f9-d8cf55c8da4c", kill_on_drop: false }
[15:58:23.031] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-70f6988a-3420-43c2-b0f9-d8cf55c8da4c", kill_on_drop: false }
[15:58:23.031] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-70f6988a-3420-43c2-b0f9-d8cf55c8da4c", kill_on_drop: false }
[15:58:23.031] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-70f6988a-3420-43c2-b0f9-d8cf55c8da4c", kill_on_drop: false }
[15:58:23.031] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.031] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.031] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.032] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.035] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.035] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.035] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.036] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.036] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.036] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.036] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.036] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.037] [server] Server setup complete
[15:58:23.037] [server] Server setup complete
[15:58:23.037] [server] Server setup complete
[15:58:23.037] [server] Server setup complete
[15:58:23.037] [server] Starting server...
[15:58:23.037] [server] Starting server...
[15:58:23.038] [server] Starting server...
[15:58:23.038] [server] Starting server...
[15:58:23.038] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-f3d34249-8f7e-431a-ad42-6b58e37bc9d0", kill_on_drop: false }
[15:58:23.038] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-f3d34249-8f7e-431a-ad42-6b58e37bc9d0", kill_on_drop: false }
[15:58:23.038] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-f3d34249-8f7e-431a-ad42-6b58e37bc9d0", kill_on_drop: false }
[15:58:23.038] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-f3d34249-8f7e-431a-ad42-6b58e37bc9d0", kill_on_drop: false }
[15:58:23.132] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.132] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.132] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.132] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.135] ------




[15:58:23.135] SSH Resolver called for "ssh-remote+remote-server.internal", attempt 5, (Reconnection)
[15:58:23.137] SSH Resolver called for host: remote-server.internal
[15:58:23.137] Setting up SSH remote "remote-server.internal"
[15:58:23.138] Acquiring local install lock: /var/folders/yx/3q47twms4m91vr48phzmhnym0000gn/T/vscode-remote-ssh-57434044-install.lock
[15:58:23.138] Looking for existing server data file at /Users/user/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-57434044-e170252f762678dec6ca2cc69aba1570769a5d39-0.110.1-es/data.json
[15:58:23.139] Found local server running: {"remoteListeningOn":{"port":40253},"osReleaseId":"debian","arch":"x86_64","display":"","tmpDir":"/tmp","platform":"linux","execServerToken":"b3001bb7-1a0b-4a2a-8ff3-1444f541fc2e","pid":51884,"ipcHandlePath":"/var/folders/yx/3q47twms4m91vr48phzmhnym0000gn/T/vscode-ssh-askpass-932b3f9e8d5f05298b1d31a1773bc9a265fe5455.sock","socksPort":63878,"startupTime":1713913100387}
[15:58:23.139] Running server is stale. Ignoring
[15:58:23.139] Using commit id "e170252f762678dec6ca2cc69aba1570769a5d39" and quality "stable" for server
[15:58:23.142] Install and start server if needed
[15:58:23.146] askpass server listening on /var/folders/yx/3q47twms4m91vr48phzmhnym0000gn/T/vscode-ssh-askpass-3c13d5406dbc2567d282478999fc14b44d1c7ab9.sock
[15:58:23.146] Spawning local server with {"serverId":4,"ipcHandlePath":"/var/folders/yx/3q47twms4m91vr48phzmhnym0000gn/T/vscode-ssh-askpass-fb8a22db67fcf398ed895bfa5d94680f0525ae39.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","63898","-o","ConnectTimeout=15","remote-server.internal"],"serverDataFolderName":".vscode-server","dataFilePath":"/Users/user/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-57434044-e170252f762678dec6ca2cc69aba1570769a5d39-0.110.1-es/data.json"}
[15:58:23.146] Local server env: {"SSH_AUTH_SOCK":"/private/tmp/com.apple.launchd.8LTFXG9Hr1/Listeners","SHELL":"/bin/zsh","DISPLAY":"/private/tmp/com.apple.launchd.pERavFJZNJ/org.xquartz:0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.110.1/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin)","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"","VSCODE_SSH_ASKPASS_MAIN":"/Users/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.110.1/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/yx/3q47twms4m91vr48phzmhnym0000gn/T/vscode-ssh-askpass-3c13d5406dbc2567d282478999fc14b44d1c7ab9.sock"}
[15:58:23.147] Spawned 51907
[15:58:23.270] > local-server-4> Running ssh connection command: ssh -v -T -D 63898 -o ConnectTimeout=15 remote-server.internal
[15:58:23.271] > local-server-4> Spawned ssh, pid=51912
[15:58:23.281] stderr> OpenSSH_9.6p1, LibreSSL 3.3.6
[15:58:23.412] stderr> found ussh cert valid for 15h28m36s
[15:58:24.480] stderr> debug1: Server host key: ssh-ed25519 SHA256:0LKvuoVLkEA5+6nLIMqnu6QTe14NTT6NaFGZsOrSg0c
[15:58:24.482] stderr> Warning: Permanently added 'remote-server.internal' (ED25519) to the list of known hosts.
[15:58:24.761] > local-server-2> Timed out
[15:58:24.773] Local server exit: 0
[15:58:24.906] stderr> Authenticated to remote-server.internal (via proxy) using "publickey".
[15:58:25.107] >

useLocalServer: false -> hangs

[15:59:33.930] Log Level: 2
[15:59:33.935] VS Code version: 1.88.1
[15:59:33.935] Remote-SSH version: remote-ssh@0.110.1
[15:59:33.935] darwin arm64
[15:59:33.937] SSH Resolver called for "ssh-remote+remote-server.internal", attempt 1
[15:59:33.937] "remote.SSH.useLocalServer": false
[15:59:33.937] "remote.SSH.useExecServer": true
[15:59:33.937] "remote.SSH.showLoginTerminal": false
[15:59:33.937] "remote.SSH.remotePlatform": {}
[15:59:33.937] "remote.SSH.path": undefined
[15:59:33.937] "remote.SSH.configFile": undefined
[15:59:33.937] "remote.SSH.useFlock": true
[15:59:33.938] "remote.SSH.lockfilesInTmp": false
[15:59:33.938] "remote.SSH.localServerDownload": auto
[15:59:33.938] "remote.SSH.remoteServerListenOnSocket": false
[15:59:33.938] "remote.SSH.showLoginTerminal": false
[15:59:33.939] "remote.SSH.defaultExtensions": []
[15:59:33.939] "remote.SSH.loglevel": 2
[15:59:33.939] "remote.SSH.enableDynamicForwarding": true
[15:59:33.939] "remote.SSH.enableRemoteCommand": false
[15:59:33.939] "remote.SSH.serverPickPortsFromRange": {}
[15:59:33.939] "remote.SSH.serverInstallPath": {}
[15:59:33.943] SSH Resolver called for host: remote-server.internal
[15:59:33.943] Setting up SSH remote "remote-server.internal"
[15:59:33.944] Using commit id "e170252f762678dec6ca2cc69aba1570769a5d39" and quality "stable" for server
[15:59:33.946] Install and start server if needed

Expected Behavior

Expectation is that when something like this happens (remote installation corrupted), VS Code should try to reinstall the remote installation.

Steps To Reproduce

Hard to say why the install got corrupted, but did see it with two of our users already. Disabling useExecServer worked, but I'm guessing that uses a whole separate flow.

Remote-SSH Log

Remote-SSH Log 

[15:58:22.613] > ready: 39ca82d963d2
[15:58:22.654] > Linux 5.10.197+ #1 SMP Fri Dec 29 09:13:24 UTC 2023
[15:58:22.655] Platform: linux
[15:58:22.693] > /bin/zsh
[15:58:22.694] Shell: zsh
[15:58:22.694] Creating bash subshell inside "zsh"
[15:58:22.804] > bash version:  5.0.3(1)-release
[15:58:22.841] > bash version:  5.0.3(1)-release
> bash version:  5.0.3(1)-release
[15:58:22.861] > 39ca82d963d2: running
> Found existing installation at /home/user/.vscode-server...
> Starting VS Code CLI... "/home/user/.vscode-server/code-e170252f762678dec6ca2cc69aba1570769a5d39" command-shell --cli-data-dir "/home/user/.vscode-server/cli" --on-port --parent-process-id 391372 &> "/home/user/.vscode-server/.cli.e170252f762678dec6ca2cc69aba1570769a5d39.log" < /dev/null
> printenv:
[15:58:22.877] >     SHELL=/bin/zsh
>     DOCKER_TLS_VERIFY=1
>     NVM_INC=/opt/nodejs/.nvm/versions/node/v16.15.0/include/node
>     JAVA_HOME=/usr/lib/jvm/temurin-8-jdk-amd64
>     SSH_AUTH_SOCK=/var/lib/remotepod/ssh/active_ssh_auth_sock
>     REMOTEPOD_USER_FULL_NAME=
>     REMOTEPOD_NAME=user
>     PWD=/home/user
>     KUBERNETES_REMOTEPOD_HOST_IP=10.138.0.41
>     LOGNAME=user
>     UBAZEL_PROFILE_ACTIONS=critical_path
>     REMOTEPOD_SHELL=/bin/zsh
>     REMOTEPOD_IDE_CONNECTED_CLIENTS=/opt/gateway/connected_clients
>     ORG_JETBRAINS_PROJECTOR_SERVER_ENABLE_WS_TRANSPORT=true
>     KUBERNETES_REMOTEPOD_CPU_LIMIT=48
>     HOME=/home/user
>     REMOTEPOD_ENVIRONMENT=kubernetes
>     CWM_TRUST_PROJECTS=1
>     LDA_HOME=/opt/lda
>     TMPDIR=/tmp
>     VSCODE_AGENT_FOLDER=/home/user/.vscode-server
>     SSH_CONNECTION=127.16.150.9 34268 127.17.145.32 22
>     REMOTEPOD_TELEGRAF_PORT_HTTP=49090
>     REMOTEPOD_RELEASE_CHANNEL=stable
>     NVM_DIR=/opt/nodejs/.nvm
>     REMOTEPOD_DISK_SIZE=256Gi
>     KUBERNETES_REMOTEPOD_MEM_LIMIT=96
>     UBAZEL_PROFILE_SAMPLING=100
>     KUBERNETES_REMOTEPOD_CPU_REQUEST=2
>     USER=user
>     DOCKER_CERT_PATH=/home/user/.docker
>     NPM_CONFIG_GLOBALCONFIG=/etc/npmrc
>     SHLVL=1
>     NVM_CD_FLAGS=-q
>     REMOTEPOD_METRICS_PATH=/var/lib/remotepod/metrics
>     KUBERNETES_REMOTEPOD_POD_IP=127.17.145.32
>     KUBERNETES_REMOTEPOD_POD_NAME=user-865744db97-tbwv5
>     KUBERNETES_REMOTEPOD_MEM_REQUEST=16
>     GO111MODULE=on
>     SSH_CLIENT=10.16.150.9 34268 22
>     LC_ALL=en_US.UTF-8
>     PATH=/opt/nodejs/.nvm/versions/node/v16.15.0/bin:/opt/lda/pex_resources/scripts/binaries:/home/user/bin:/opt/git/bin:/home/user/go-repos:/opt/go/path/bin:/opt/go/root/bin:/usr/lib/jvm/temurin-8-jdk-amd64/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/nodejs/node_modules/bin
>     KUBERNETES_REMOTEPOD_NODE_NAME=gke-remotepod-remotepod-1-24-highmem-af79f649-7vws
>     REMOTEPOD_VERSION=0.4.58
>     REMOTEPOD_FLAVOR=go
>     NVM_BIN=/opt/nodejs/.nvm/versions/node/v16.15.0/bin
>     MAIL=/var/mail/user
>     OLDPWD=/home/user
>     GOPATH=/home/user/go-repos:/opt/go/path
>     _=/usr/bin/printenv
> Removing old logfile at /home/user/.vscode-server/.cli.e170252f762678dec6ca2cc69aba1570769a5d39.log
> Spawned remote CLI: 391392
> Waiting for server log...
[15:58:22.912] > 39ca82d963d2: start
> DISPLAY====
> listeningOn==127.0.0.1:40253==
> osReleaseId==debian==
> arch==x86_64==
> vscodeArch==x64==
> bitness==64==
> tmpDir==/tmp==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
> serverStartTime==45==
> execServerToken==a1111aa1-1a1a-1a1a-1aa1-1111a111aa1a==
> 39ca82d963d2: end
[15:58:22.913] Received install output:
DISPLAY====
listeningOn==127.0.0.1:40253==
osReleaseId==debian==
arch==x86_64==
vscodeArch==x64==
bitness==64==
tmpDir==/tmp==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
serverStartTime==45==
execServerToken==a1111aa1-1a1a-1a1a-1aa1-1111a111aa1a==

[15:58:22.913] Remote server is listening on port 40253
[15:58:22.913] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":40253},"osReleaseId":"debian","arch":"x86_64","display":"","tmpDir":"/tmp","platform":"linux","execServerToken":"a1111aa1-1a1a-1a1a-1aa1-1111a111aa1a"},"serverStartTime":45,"installUnpackCode":""}
[15:58:22.914] Persisting server connection details to /Users/user/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-57434044-e170252f762678dec6ca2cc69aba1570769a5d39-0.110.1-es/data.json
[15:58:22.919] Starting forwarding server. local port 63895 -> socksPort 63878 -> remotePort 40253
[15:58:22.920] Forwarding server listening on port 63895
[15:58:22.920] Waiting for ssh tunnel to be ready
[15:58:22.921] [Forwarding server port 63895] Got connection 0
[15:58:22.922] Tunneled port 40253 to local port 63895
[15:58:22.922] Resolved "ssh-remote+remote-server.internal" to "port 63895"
[15:58:22.981] Verified and reusing cached exec server for ssh-remote+remote-server.internal
[15:58:22.982] ------




[15:58:23.024] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.024] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.024] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.024] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.027] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.028] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.028] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.028] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.029] [server] Server setup complete
[15:58:23.029] [server] Server setup complete
[15:58:23.029] [server] Server setup complete
[15:58:23.029] [server] Server setup complete
[15:58:23.030] [server] Starting server...
[15:58:23.030] [server] Starting server...
[15:58:23.030] [server] Starting server...
[15:58:23.030] [server] Starting server...
[15:58:23.030] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-70f6988a-3420-43c2-b0f9-d8cf55c8da4c", kill_on_drop: false }
[15:58:23.031] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-70f6988a-3420-43c2-b0f9-d8cf55c8da4c", kill_on_drop: false }
[15:58:23.031] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-70f6988a-3420-43c2-b0f9-d8cf55c8da4c", kill_on_drop: false }
[15:58:23.031] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-70f6988a-3420-43c2-b0f9-d8cf55c8da4c", kill_on_drop: false }
[15:58:23.031] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.031] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.031] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.032] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.035] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.035] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.035] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.036] [server] Checking /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/log.txt and /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/pid.txt for a running server...
[15:58:23.036] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.036] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.036] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.036] [server] Installing and setting up Visual Studio Code Server...
[15:58:23.037] [server] Server setup complete
[15:58:23.037] [server] Server setup complete
[15:58:23.037] [server] Server setup complete
[15:58:23.037] [server] Server setup complete
[15:58:23.037] [server] Starting server...
[15:58:23.037] [server] Starting server...
[15:58:23.038] [server] Starting server...
[15:58:23.038] [server] Starting server...
[15:58:23.038] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-f3d34249-8f7e-431a-ad42-6b58e37bc9d0", kill_on_drop: false }
[15:58:23.038] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-f3d34249-8f7e-431a-ad42-6b58e37bc9d0", kill_on_drop: false }
[15:58:23.038] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-f3d34249-8f7e-431a-ad42-6b58e37bc9d0", kill_on_drop: false }
[15:58:23.038] [server] Starting server with command... Command { std: "/home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server" "--connection-token=remotessh" "--accept-server-license-terms" "--start-server" "--enable-remote-auto-shutdown" "--socket-path=/tmp/code-f3d34249-8f7e-431a-ad42-6b58e37bc9d0", kill_on_drop: false }
[15:58:23.132] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.132] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.132] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.132] [server] /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: 12: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/bin/code-server: /home/user/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node: not found
[15:58:23.135] ------




[15:58:23.135] SSH Resolver called for "ssh-remote+remote-server.internal", attempt 5, (Reconnection)
[15:58:23.137] SSH Resolver called for host: remote-server.internal
[15:58:23.137] Setting up SSH remote "remote-server.internal"
[15:58:23.138] Acquiring local install lock: /var/folders/yx/3q47twms4m91vr48phzmhnym0000gn/T/vscode-remote-ssh-57434044-install.lock
[15:58:23.138] Looking for existing server data file at /Users/user/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-57434044-e170252f762678dec6ca2cc69aba1570769a5d39-0.110.1-es/data.json
[15:58:23.139] Found local server running: {"remoteListeningOn":{"port":40253},"osReleaseId":"debian","arch":"x86_64","display":"","tmpDir":"/tmp","platform":"linux","execServerToken":"b3001bb7-1a0b-4a2a-8ff3-1444f541fc2e","pid":51884,"ipcHandlePath":"/var/folders/yx/3q47twms4m91vr48phzmhnym0000gn/T/vscode-ssh-askpass-932b3f9e8d5f05298b1d31a1773bc9a265fe5455.sock","socksPort":63878,"startupTime":1713913100387}
[15:58:23.139] Running server is stale. Ignoring
[15:58:23.139] Using commit id "e170252f762678dec6ca2cc69aba1570769a5d39" and quality "stable" for server
[15:58:23.142] Install and start server if needed
[15:58:23.146] askpass server listening on /var/folders/yx/3q47twms4m91vr48phzmhnym0000gn/T/vscode-ssh-askpass-3c13d5406dbc2567d282478999fc14b44d1c7ab9.sock
[15:58:23.146] Spawning local server with {"serverId":4,"ipcHandlePath":"/var/folders/yx/3q47twms4m91vr48phzmhnym0000gn/T/vscode-ssh-askpass-fb8a22db67fcf398ed895bfa5d94680f0525ae39.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","63898","-o","ConnectTimeout=15","remote-server.internal"],"serverDataFolderName":".vscode-server","dataFilePath":"/Users/user/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-57434044-e170252f762678dec6ca2cc69aba1570769a5d39-0.110.1-es/data.json"}
[15:58:23.146] Local server env: {"SSH_AUTH_SOCK":"/private/tmp/com.apple.launchd.8LTFXG9Hr1/Listeners","SHELL":"/bin/zsh","DISPLAY":"/private/tmp/com.apple.launchd.pERavFJZNJ/org.xquartz:0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.110.1/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin)","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"","VSCODE_SSH_ASKPASS_MAIN":"/Users/user/.vscode/extensions/ms-vscode-remote.remote-ssh-0.110.1/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/yx/3q47twms4m91vr48phzmhnym0000gn/T/vscode-ssh-askpass-3c13d5406dbc2567d282478999fc14b44d1c7ab9.sock"}
[15:58:23.147] Spawned 51907
[15:58:23.270] > local-server-4> Running ssh connection command: ssh -v -T -D 63898 -o ConnectTimeout=15 remote-server.internal
[15:58:23.271] > local-server-4> Spawned ssh, pid=51912
[15:58:23.281] stderr> OpenSSH_9.6p1, LibreSSL 3.3.6
[15:58:23.412] stderr> found ussh cert valid for 15h28m36s
[15:58:24.480] stderr> debug1: Server host key: ssh-ed25519 SHA256:0LKvuoVLkEA5+6nLIMqnu6QTe14NTT6NaFGZsOrSg0c
[15:58:24.482] stderr> Warning: Permanently added 'remote-server.internal' (ED25519) to the list of known hosts.
[15:58:24.761] > local-server-2> Timed out
[15:58:24.773] Local server exit: 0
[15:58:24.906] stderr> Authenticated to remote-server.internal (via proxy) using "publickey".
[15:58:25.107] >

Anything else?

No response
@JamyDev JamyDev added the ssh Issue in vscode-remote SSH label Apr 24, 2024
@roblourens roblourens assigned connor4312 and unassigned roblourens Apr 26, 2024
@roblourens
Copy link
Member

Does it work if you disable the useExecServer settting?

@roblourens roblourens reopened this Apr 26, 2024
connor4312 added a commit to microsoft/vscode that referenced this issue Apr 26, 2024
@connor4312
cli: automatically remove servers that fail to start
b15d62e 
Fixes microsoft/vscode-remote-release#9823
@connor4312 connor4312 mentioned this issue Apr 26, 2024
Merged
@connor4312 connor4312 added this to the May 2024 milestone Apr 26, 2024
@connor4312 connor4312 added the bug Issue identified by VS Code Team member as probable bug label Apr 26, 2024
connor4312 added a commit to microsoft/vscode that referenced this issue Apr 27, 2024
@connor4312
cli: automatically remove servers that fail to start (#211500)
0f2b535 
Fixes microsoft/vscode-remote-release#9823
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@connor4312 connor4312

Labels
bug Issue identified by VS Code Team member as probable bug ssh Issue in vscode-remote SSH
Projects
None yet
Milestone
May 2024
Development

Successfully merging a pull request may close this issue.

cli: automatically remove servers that fail to start microsoft/vscode
3 participants
@roblourens @JamyDev @connor4312