Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Connection resets breaking synchronization #505

Open
dindurthy opened this issue Feb 24, 2021 · 0 comments
Open

Connection resets breaking synchronization #505

dindurthy opened this issue Feb 24, 2021 · 0 comments

Comments

@dindurthy
Copy link

dindurthy commented Feb 24, 2021 •
edited

Howdy. We've had quite a bit of success with ksync, but this one is stymieing us. Any help would be appreciated.

We are running an EKS cluster initialized by ksync. It's a shared cluster and most of my colleagues are using ksync with no problems. However, I have one engineer (so far) facing this.

TLDR the ksync watch process is stuck logging updating due to connection resets and the pod never syncs.

% ksync version
ksync:
	Version:    Release
	Go Version: go1.14.15
	Git Commit: bfb445b
	Git Tag:    0.4.6
	Built:      Wed Feb 10 22:49:20 +0000 2021
	OS/Arch:    darwin/amd64
service:
	Version:    Release
	Go Version: go1.14.4
	Git Commit: 9f40bf1
	Git Tag:    0.4.5
	Built:      Tue Jun 30 06:40:33 +0000 2020

Here's the debug log of the watch process:

DEBU[0000] initializing kubernetes client                context=
DEBU[0000] kubernetes client created                     host="https://****.gr7.us-east-1.eks.amazonaws.com"
DEBU[0001] watching for updates                          ContainerName= LocalPath=/Users/tc/code/redox-services/services/archt-345-poc-kafka-distributed-cache LocalReadOnly=true Name=archt-345-poc-kafka-distributed-cache-app Namespace=tc Pod= Reload=true RemotePath=/opt/nodejs RemoteReadOnly=false Selector="[service=archt-345-poc-kafka-distributed-cache ksync=app]"
DEBU[0001] watching for updates                          ContainerName= LocalPath=/Users/tc/code/redox-services/services/archt-345-poc-kafka-distributed-cache LocalReadOnly=true Name=archt-345-poc-kafka-distributed-cache-cache-user Namespace=tc Pod= Reload=true RemotePath=/opt/nodejs RemoteReadOnly=false Selector="[service=archt-345-poc-kafka-distributed-cache ksync=cache-user]"
DEBU[0001] new event                                     deleted=false name=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv status=Running type=ADDED
INFO[0001] new pod detected                              pod=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv spec=archt-345-poc-kafka-distributed-cache-cache-user
DEBU[0001] added service                                 ID=4fc34fc34491d34f41c64e1a1d9f95778057b027f8551f18029155f27b115981 Name=archt-345-poc-kafka-distributed-cache-user NodeName=ip-172-31-90-253.ec2.internal PodName=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv
DEBU[0001] checking to see if radar is ready             nodeName=ip-172-31-90-253.ec2.internal
DEBU[0001] cleaning background daemon                   
DEBU[0001] starting syncthing                            args="[/Users/tc/.ksync/bin/syncthing -gui-address localhost:8384 -gui-apikey <uuid> -home /Users/tc/.ksync/syncthing -no-browser]" cmd=/Users/tc/.ksync/bin/syncthing
INFO[0001] listening                                     bind=127.0.0.1 port=40322
DEBU[0001] RESTY 2021/02/24 13:03:24 ERROR Get "http://localhost:8384/rest/system/config": dial tcp [::1]:8384: connect: connection refused, Attempt 1 
DEBU[0002] found pod name                                Namespace=kube-system RadarPort=40321 SyncthingAPI=8384 SyncthingListener=22000 nodeName=ip-172-31-90-253.ec2.internal podName=ksync-9bdlz
DEBU[0002] found pod                                     nodeName=ip-172-31-90-253.ec2.internal podName=ksync-9bdlz status=Running
DEBU[0002] starting tunnel                               LocalPort=64393 Namespace=kube-system Out= PodName=ksync-9bdlz RemotePort=40321 url="https://***.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/kube-system/pods/ksync-9bdlz/portforward"
DEBU[0002] [start] 13:03:25 INFO: syncthing v1.13.1 "Fermium Flea" (go1.15.7 darwin-amd64) teamcity@build.syncthing.net 2021-01-11 14:15:21 UTC  name=syncthing
DEBU[0002] [start] 13:03:25 INFO: Generating ECDSA key and certificate for syncthing...  name=syncthing
DEBU[0002] [start] 13:03:25 INFO: Archiving a copy of old config file format at: /Users/tc/.ksync/syncthing/config.xml.v26  name=syncthing
DEBU[0002] [B7CY5] 13:03:25 INFO: My ID: <device id>  name=syncthing
DEBU[0002] RESTY 2021/02/24 13:03:25 ERROR Get "http://localhost:8384/rest/system/config": dial tcp [::1]:8384: connect: connection refused, Attempt 2 
DEBU[0003] tunnel running                                LocalPort=64393 Namespace=kube-system Out="Forwarding from 127.0.0.1:64393 -> 40321\nForwarding from [::1]:64393 -> 40321\n" PodName=ksync-9bdlz RemotePort=40321
DEBU[0003] [B7CY5] 13:03:26 INFO: Single thread SHA256 performance is 428 MB/s using minio/sha256-simd (380 MB/s using crypto/sha256).  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Hashing performance is 357.71 MB/s  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 1...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 2...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 3...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 5...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 6...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 7...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 9...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 10...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 11...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 13...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 14...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Running database migration 15...  name=syncthing
DEBU[0004] [B7CY5] 13:03:26 INFO: Compacting database after migration...  name=syncthing
DEBU[0004] [B7CY5] 13:03:27 INFO: Overall send rate is unlimited, receive rate is unlimited  name=syncthing
DEBU[0004] [B7CY5] 13:03:27 INFO: Loading HTTPS certificate: open /Users/tc/.ksync/syncthing/https-cert.pem: no such file or directory  name=syncthing
DEBU[0004] [B7CY5] 13:03:27 INFO: Creating new HTTPS certificate  name=syncthing
DEBU[0004] [B7CY5] 13:03:27 INFO: TCP listener ([::]:22000) starting  name=syncthing
DEBU[0004] 2021/02/24 13:03:27 connection doesn't allow setting of receive buffer size. See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.  name=syncthing
DEBU[0004] [B7CY5] 13:03:27 INFO: QUIC listener ([::]:22000) starting  name=syncthing
DEBU[0004] [B7CY5] 13:03:27 INFO: GUI and API listening on 127.0.0.1:8384  name=syncthing
DEBU[0004] [B7CY5] 13:03:27 INFO: Access the GUI via the following URL: http://localhost:8384/  name=syncthing
DEBU[0004] [B7CY5] 13:03:27 INFO: My name is "ip-10-0-0-37.us-east-2.compute.internal"  name=syncthing
INFO[0004] syncthing listening                           port=8384 syncthing=localhost
DEBU[0005] checking to see if radar is ready             nodeName=ip-172-31-90-253.ec2.internal
DEBU[0005] found pod name                                Namespace=kube-system RadarPort=40321 SyncthingAPI=8384 SyncthingListener=22000 nodeName=ip-172-31-90-253.ec2.internal podName=ksync-9bdlz
DEBU[0005] found pod                                     nodeName=ip-172-31-90-253.ec2.internal podName=ksync-9bdlz status=Running
DEBU[0006] starting tunnel                               LocalPort=64407 Namespace=kube-system Out= PodName=ksync-9bdlz RemotePort=8384 url="https://***.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/kube-system/pods/ksync-9bdlz/portforward"
DEBU[0006] tunnel running                                LocalPort=64407 Namespace=kube-system Out="Forwarding from 127.0.0.1:64407 -> 8384\nForwarding from [::1]:64407 -> 8384\n" PodName=ksync-9bdlz RemotePort=8384
DEBU[0006] checking to see if radar is ready             nodeName=ip-172-31-90-253.ec2.internal
DEBU[0006] found pod name                                Namespace=kube-system RadarPort=40321 SyncthingAPI=8384 SyncthingListener=22000 nodeName=ip-172-31-90-253.ec2.internal podName=ksync-9bdlz
DEBU[0006] found pod                                     nodeName=ip-172-31-90-253.ec2.internal podName=ksync-9bdlz status=Running
DEBU[0007] starting tunnel                               LocalPort=64409 Namespace=kube-system Out= PodName=ksync-9bdlz RemotePort=22000 url="https://***.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/kube-system/pods/ksync-9bdlz/portforward"
DEBU[0007] tunnel running                                LocalPort=64409 Namespace=kube-system Out="Forwarding from 127.0.0.1:64409 -> 22000\nForwarding from [::1]:64409 -> 22000\n" PodName=ksync-9bdlz RemotePort=22000
DEBU[0008] [B7CY5] 13:03:30 INFO: Adding folder "archt-345-poc-kafka-distributed-cache-cache-user-archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv" (archt-345-poc-kafka-distributed-cache-cache-user-archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv)  name=syncthing
DEBU[0008] [B7CY5] 13:03:30 INFO: No stored folder metadata for "archt-345-poc-kafka-distributed-cache-cache-user-archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv"; recalculating  name=syncthing
DEBU[0008] [B7CY5] 13:03:30 INFO: Ready to synchronize "archt-345-poc-kafka-distributed-cache-cache-user-archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv" (archt-345-poc-kafka-distributed-cache-cache-user-archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv) (sendonly)  name=syncthing
WARN[0008] {
  "alloc": 41094832,
  "connectionServiceStatus": {
    "quic://0.0.0.0:22000": {
      "error": null,
      "lanAddresses": [
        "quic://0.0.0.0:22000",
        "quic://10.0.0.37:22000",
        "quic://172.27.236.89:22000"
      ],
      "wanAddresses": [
        "quic://0.0.0.0:22000"
      ]
    },
    "tcp://0.0.0.0:22000": {
      "error": null,
      "lanAddresses": [
        "tcp://0.0.0.0:22000",
        "tcp://10.0.0.37:22000",
        "tcp://172.27.236.89:22000"
      ],
      "wanAddresses": [
        "tcp://0.0.0.0:0",
        "tcp://0.0.0.0:22000"
      ]
    }
  },
  "cpuPercent": 0,
  "goroutines": 58,
  "guiAddressOverridden": true,
  "guiAddressUsed": "127.0.0.1:8384",
  "lastDialStatus": {},
  "myID": "<device id>",
  "pathSeparator": "/",
  "startTime": "2021-02-24T13:03:25.294297-07:00",
  "sys": 53273600,
  "tilde": "/Users/tc",
  "uptime": 5,
  "urVersionMax": 3
} 
WARN[0008] {
  "alloc": 41191280,
  "connectionServiceStatus": {
    "quic://0.0.0.0:22000": {
      "error": null,
      "lanAddresses": [
        "quic://0.0.0.0:22000",
        "quic://10.0.0.37:22000",
        "quic://172.27.236.89:22000"
      ],
      "wanAddresses": [
        "quic://0.0.0.0:22000"
      ]
    },
    "tcp://0.0.0.0:22000": {
      "error": null,
      "lanAddresses": [
        "tcp://0.0.0.0:22000",
        "tcp://10.0.0.37:22000",
        "tcp://172.27.236.89:22000"
      ],
      "wanAddresses": [
        "tcp://0.0.0.0:0",
        "tcp://0.0.0.0:22000"
      ]
    }
  },
  "cpuPercent": 0,
  "goroutines": 60,
  "guiAddressOverridden": true,
  "guiAddressUsed": "127.0.0.1:8384",
  "lastDialStatus": {},
  "myID": "<device id>",
  "pathSeparator": "/",
  "startTime": "2021-02-24T13:03:25.294297-07:00",
  "sys": 53322752,
  "tilde": "/Users/tc",
  "uptime": 5,
  "urVersionMax": 3
} 
DEBU[0008] restarting local syncthing                   
INFO[0008] finished unary call with code OK              grpc.code=OK grpc.method=RestartSyncthing grpc.service=proto.ksync.Ksync grpc.start_time="2021-02-24T13:03:31-07:00" grpc.time_ns=1119512 peer.address="127.0.0.1:64399" span.kind=server system=grpc
INFO[0008] folder sync running                           pod=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv spec=archt-345-poc-kafka-distributed-cache-cache-user
DEBU[0008] RESTY 2021/02/24 13:03:31 ERROR Get "http://localhost:8384/rest/events?since=8": dial tcp [::1]:8384: connect: connection refused, Attempt 1 
DEBU[0008] [B7CY5] 13:03:31 INFO: Completed initial scan of sendonly folder "archt-345-poc-kafka-distributed-cache-cache-user-archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv" (archt-345-poc-kafka-distributed-cache-cache-user-archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv)  name=syncthing
DEBU[0008] [B7CY5] 13:03:31 INFO: QUIC listener ([::]:22000) shutting down  name=syncthing
DEBU[0009] RESTY 2021/02/24 13:03:32 ERROR Get "http://localhost:8384/rest/events?since=8": dial tcp [::1]:8384: connect: connection refused, Attempt 2 
DEBU[0009] [B7CY5] 13:03:32 INFO: TCP listener ([::]:22000) shutting down  name=syncthing
DEBU[0009] [B7CY5] 13:03:32 INFO: Exiting                name=syncthing
DEBU[0009] [monitor] 13:03:32 INFO: Syncthing exited: exit status 3  name=syncthing
DEBU[0010] [start] 13:03:33 INFO: syncthing v1.13.1 "Fermium Flea" (go1.15.7 darwin-amd64) teamcity@build.syncthing.net 2021-01-11 14:15:21 UTC  name=syncthing
DEBU[0010] [B7CY5] 13:03:33 INFO: My ID: <device id>  name=syncthing
DEBU[0010] RESTY 2021/02/24 13:03:33 ERROR Get "http://localhost:8384/rest/events?since=8": dial tcp [::1]:8384: connect: connection refused, Attempt 3 
DEBU[0011] [B7CY5] 13:03:34 INFO: Single thread SHA256 performance is 474 MB/s using minio/sha256-simd (413 MB/s using crypto/sha256).  name=syncthing
DEBU[0012] [B7CY5] 13:03:34 INFO: Hashing performance is 389.35 MB/s  name=syncthing
DEBU[0012] [B7CY5] 13:03:34 INFO: Overall send rate is unlimited, receive rate is unlimited  name=syncthing
DEBU[0012] [B7CY5] 13:03:34 INFO: TCP listener ([::]:22000) starting  name=syncthing
DEBU[0012] 2021/02/24 13:03:34 connection doesn't allow setting of receive buffer size. See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.  name=syncthing
DEBU[0012] [B7CY5] 13:03:34 INFO: Ready to synchronize "archt-345-poc-kafka-distributed-cache-cache-user-archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv" (archt-345-poc-kafka-distributed-cache-cache-user-archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv) (sendonly)  name=syncthing
DEBU[0012] [B7CY5] 13:03:34 INFO: QUIC listener ([::]:22000) starting  name=syncthing
DEBU[0012] [B7CY5] 13:03:34 INFO: Completed initial scan of sendonly folder "archt-345-poc-kafka-distributed-cache-cache-user-archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv" (archt-345-poc-kafka-distributed-cache-cache-user-archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv)  name=syncthing
DEBU[0012] [B7CY5] 13:03:34 INFO: GUI and API listening on 127.0.0.1:8384  name=syncthing
DEBU[0012] [B7CY5] 13:03:34 INFO: Access the GUI via the following URL: http://localhost:8384/  name=syncthing
DEBU[0012] [B7CY5] 13:03:34 INFO: My name is "ip-10-0-0-37.us-east-2.compute.internal"  name=syncthing
DEBU[0012] [B7CY5] 13:03:34 INFO: Device <device id> is "archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv" at [tcp://127.0.0.1:64409]  name=syncthing
DEBU[0012] [B7CY5] 13:03:35 INFO: Established secure connection to <device id> at 127.0.0.1:22000-127.0.0.1:64409/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256  name=syncthing
DEBU[0012] [B7CY5] 13:03:35 INFO: Device <device id> client is "syncthing v1.6.1" named "" at 127.0.0.1:22000-127.0.0.1:64409/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256  name=syncthing
DEBU[0012] error copying from local connection to remote stream: read tcp4 127.0.0.1:64409->127.0.0.1:22000: read: connection reset by peer 
DEBU[0012] lost connection to remote                     node=ip-172-31-90-253.ec2.internal pod=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv
DEBU[0012] [B7CY5] 13:03:35 INFO: Connection to <device id> at 127.0.0.1:22000-127.0.0.1:64409/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256 closed: reading length: EOF  name=syncthing
DEBU[0012] an error occurred forwarding 64409 -> 22000: error forwarding port 22000 to pod 41e312b2d482368457d83d6f769df677fe7daf93882c9b0d222b255f0b2296be, uid : exit status 1: 2021/02/24 20:03:35 socat[6040] E write(5, 0x102d120, 286): Broken pipe 
DEBU[0012] lost connection to remote                     node=ip-172-31-90-253.ec2.internal pod=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv
INFO[0014] updating                                      pod=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv spec=archt-345-poc-kafka-distributed-cache-cache-user
DEBU[0017] [B7CY5] 13:03:40 INFO: Established secure connection to <device id> at 127.0.0.1:22000-127.0.0.1:64409/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256  name=syncthing
DEBU[0017] [B7CY5] 13:03:40 INFO: Device <device id> client is "syncthing v1.6.1" named "" at 127.0.0.1:22000-127.0.0.1:64409/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256  name=syncthing
DEBU[0017] error copying from local connection to remote stream: read tcp4 127.0.0.1:64409->127.0.0.1:22000: read: connection reset by peer 
DEBU[0017] lost connection to remote                     node=ip-172-31-90-253.ec2.internal pod=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv
DEBU[0017] [B7CY5] 13:03:40 INFO: Connection to <device id> at 127.0.0.1:22000-127.0.0.1:64409/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256 closed: reading length: EOF  name=syncthing
DEBU[0017] an error occurred forwarding 64409 -> 22000: error forwarding port 22000 to pod 41e312b2d482368457d83d6f769df677fe7daf93882c9b0d222b255f0b2296be, uid : exit status 1: 2021/02/24 20:03:40 socat[7890] E write(5, 0x22eb120, 286): Broken pipe 
DEBU[0017] lost connection to remote                     node=ip-172-31-90-253.ec2.internal pod=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv
INFO[0018] updating                                      pod=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv spec=archt-345-poc-kafka-distributed-cache-cache-user
DEBU[0023] [B7CY5] 13:03:45 INFO: Established secure connection to <device id> at 127.0.0.1:22000-127.0.0.1:64409/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256  name=syncthing
DEBU[0023] [B7CY5] 13:03:45 INFO: Device <device id> client is "syncthing v1.6.1" named "" at 127.0.0.1:22000-127.0.0.1:64409/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256  name=syncthing
DEBU[0023] error copying from local connection to remote stream: read tcp4 127.0.0.1:64409->127.0.0.1:22000: read: connection reset by peer 
DEBU[0023] lost connection to remote                     node=ip-172-31-90-253.ec2.internal pod=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv
DEBU[0023] [B7CY5] 13:03:45 INFO: Connection to <device id> at 127.0.0.1:22000-127.0.0.1:64409/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256 closed: reading length: EOF  name=syncthing
DEBU[0023] an error occurred forwarding 64409 -> 22000: error forwarding port 22000 to pod 41e312b2d482368457d83d6f769df677fe7daf93882c9b0d222b255f0b2296be, uid : exit status 1: 2021/02/24 20:03:45 socat[9680] E write(5, 0x20b7120, 310): Broken pipe 
DEBU[0023] lost connection to remote                     node=ip-172-31-90-253.ec2.internal pod=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv

Basically, the log just cycles on updating. The first thing that looks awry is DEBU[0012] [B7CY5] 13:03:35 INFO: Device <device id> client is "syncthing v1.6.1" named "" at 127.0.0.1:22000-127.0.0.1:64409/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256 name=syncthing, that is, there's no device name. But I assume these logs are the real problem:

DEBU[0012] error copying from local connection to remote stream: read tcp4 127.0.0.1:64409->127.0.0.1:22000: read: connection reset by peer 
DEBU[0012] lost connection to remote                     node=ip-172-31-90-253.ec2.internal pod=archt-345-poc-kafka-distributed-cache-user-5d69f76c6d-lm2hv
DEBU[0012] [B7CY5] 13:03:35 INFO: Connection to <device id> at 127.0.0.1:22000-127.0.0.1:64409/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256 closed: reading length: EOF  name=syncthing
DEBU[0012] an error occurred forwarding 64409 -> 22000: error forwarding port 22000 to pod 41e312b2d482368457d83d6f769df677fe7daf93882c9b0d222b255f0b2296be, uid : exit status 1: 2021/02/24 20:03:35 socat[6040] E write(5, 0x102d120, 286): Broken pipe

Not exactly sure how I should be reading those. I believe that error is a kubernetes client error. I don't see anything unusual in the ksync pod logs or the application pod log. Port-forwarding in general works for this engineer. I do see log lines like this that may correlate:

Feb 23 00:36:24 ip-172-31-100-121 kubelet[2248]: E0223 00:36:24.950499    2248 httpstream.go:251] error forwarding port 22000 to pod 26e9be89345698b07a0ecce4787ffe654cdf157a4d2dc78cfd6cfd10680c7338, uid : exit status 1: 2021/02/23 00:36:24 socat[12020] E write(5, 0x2241120, 24): Broken pipe

I still can't tell from the logs which side of the connection is causing the reset. Given not every engineer is experiencing this, we started with inspecting his workstation, but we haven't found anything weird on his workstation logs.

Any help making sense of these logs would be appreciated.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dindurthy