Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A spontaneous problem with Cloak-TCP (caught a crash log this time) #230

Open
LindaFerum opened this issue Aug 11, 2023 · 5 comments
Open

Comments

@LindaFerum
Copy link

LindaFerum commented Aug 11, 2023

So long story short I have a machine in adversarial conditions using Cloak TCP

Periodically, Cloak crashes like this :

runtime: checkdead: find g 99 in status 1
fatal error: checkdead: runnable g

runtime stack:
runtime.throw({0x73b4e3?, 0x0?})
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/panic.go:1047 +0x5d fp=0x7ffeb6d52958 sp=0x7ffeb6d52928 pc=0x433e7d
runtime.checkdead.func1(0xc000007ba0)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:5216 +0xe5 fp=0x7ffeb6d52990 sp=0x7ffeb6d52958 pc=0x441a25
runtime.forEachG(0x7ffeb6d529f0)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:591 +0x5c fp=0x7ffeb6d529c0 sp=0x7ffeb6d52990 pc=0x43761c
runtime.checkdead()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:5203 +0xf7 fp=0x7ffeb6d52a10 sp=0x7ffeb6d529c0 pc=0x441717
runtime.mput(0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:5696 +0x33 fp=0x7ffeb6d52a20 sp=0x7ffeb6d52a10 pc=0x442c93
runtime.stopm()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:2332 +0x67 fp=0x7ffeb6d52a50 sp=0x7ffeb6d52a20 pc=0x43aa47
runtime.findRunnable()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:3007 +0xabc fp=0x7ffeb6d52b58 sp=0x7ffeb6d52a50 pc=0x43c2bc
runtime.schedule()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:3360 +0xb1 fp=0x7ffeb6d52b90 sp=0x7ffeb6d52b58 pc=0x43d0f1
runtime.park_m(0xc000007ba0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:3511 +0x12d fp=0x7ffeb6d52bc0 sp=0x7ffeb6d52b90 pc=0x43d60d
runtime.mcall()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:452 +0x43 fp=0x7ffeb6d52bd0 sp=0x7ffeb6d52bc0 pc=0x462fa3

goroutine 1 [IO wait, 32 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc0001256d8 sp=0xc0001256b8 pc=0x436b96
runtime.netpollblock(0x2c201223873eeccf?, 0x40404f?, 0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:527 +0xf7 fp=0xc000125710 sp=0xc0001256d8 pc=0x42f597
internal/poll.runtime_pollWait(0x7f9beda2ff78, 0x72)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89 fp=0xc000125730 sp=0xc000125710 pc=0x45ff09
internal/poll.(*pollDesc).wait(0xc0000a6180?, 0xc000357540?, 0x0)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000125758 sp=0xc000125730 pc=0x4b2f52
internal/poll.(*pollDesc).waitRead(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0000a6180)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:614 +0x2bd fp=0xc000125800 sp=0xc000125758 pc=0x4b55bd
net.(*netFD).accept(0xc0000a6180)
/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_unix.go:172 +0x35 fp=0xc0001258b8 sp=0xc000125800 pc=0x4ecaf5
net.(*TCPListener).accept(0xc000010228)
/opt/hostedtoolcache/go/1.20.4/x64/src/net/tcpsock_posix.go:148 +0x25 fp=0xc0001258e0 sp=0xc0001258b8 pc=0x4fe8c5
net.(*TCPListener).Accept(0xc000010228)
/opt/hostedtoolcache/go/1.20.4/x64/src/net/tcpsock.go:297 +0x3d fp=0xc000125910 sp=0xc0001258e0 pc=0x4fdb3d
github.com/cbeuw/Cloak/internal/client.RouteTCP({0x7b7bd0, 0xc000010228}, 0x45d964b800, 0x0, 0x1400, 0x1400, 0xc0000b80f0)
/home/runner/work/Cloak/Cloak/internal/client/piper.go:102 +0x83 fp=0xc0001259c0 sp=0xc000125910 pc=0x6b5743
main.main()
/home/runner/work/Cloak/Cloak/cmd/ck-client/ck-client.go:204 +0x1405 fp=0xc000125f80 sp=0xc0001259c0 pc=0x6bf7e5
runtime.main()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:250 +0x207 fp=0xc000125fe0 sp=0xc000125f80 pc=0x436767
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000125fe8 sp=0xc000125fe0 pc=0x4650a1

goroutine 2 [runnable]:
runtime.gopark(0xed82221cf2ae?, 0x0?, 0x0?, 0x0?, 0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000038fb0 sp=0xc000038f90 pc=0x436b96
runtime.goparkunlock(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:387
runtime.forcegchelper()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:305 +0xb0 fp=0xc000038fe0 sp=0xc000038fb0 pc=0x4369d0
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000038fe8 sp=0xc000038fe0 pc=0x4650a1
created by runtime.init.6
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:293 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000039780 sp=0xc000039760 pc=0x436b96
runtime.goparkunlock(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:387
runtime.bgsweep(0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgcsweep.go:319 +0xde fp=0xc0000397c8 sp=0xc000039780 pc=0x42325e
runtime.gcenable.func1()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:178 +0x26 fp=0xc0000397e0 sp=0xc0000397c8 pc=0x4186c6
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000397e8 sp=0xc0000397e0 pc=0x4650a1
created by runtime.gcenable
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xec16b8c0e0af?, 0x13a36492?, 0x0?, 0x0?, 0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000039f70 sp=0xc000039f50 pc=0x436b96
runtime.goparkunlock(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:387
runtime.(*scavengerState).park(0x99d3a0)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgcscavenge.go:400 +0x53 fp=0xc000039fa0 sp=0xc000039f70 pc=0x421133
runtime.bgscavenge(0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgcscavenge.go:633 +0x65 fp=0xc000039fc8 sp=0xc000039fa0 pc=0x421725
runtime.gcenable.func2()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:179 +0x26 fp=0xc000039fe0 sp=0xc000039fc8 pc=0x418666
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000039fe8 sp=0xc000039fe0 pc=0x4650a1
created by runtime.gcenable
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait, 32 minutes]:
runtime.gopark(0x436f12?, 0x7f9c14dcb868?, 0x0?, 0x0?, 0xc000038770?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000038628 sp=0xc000038608 pc=0x436b96
runtime.runfinq()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mfinal.go:193 +0x107 fp=0xc0000387e0 sp=0xc000038628 pc=0x417707
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000387e8 sp=0xc0000387e0 pc=0x4650a1
created by runtime.createfing
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mfinal.go:163 +0x45

goroutine 95 [IO wait]:
runtime.gopark(0x461bc5?, 0xb?, 0x0?, 0x0?, 0xa?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc00006ec40 sp=0xc00006ec20 pc=0x436b96
runtime.netpollblock(0x49cca5?, 0x40404f?, 0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:527 +0xf7 fp=0xc00006ec78 sp=0xc00006ec40 pc=0x42f597
internal/poll.runtime_pollWait(0x7f9beda2fac8, 0x72)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89 fp=0xc00006ec98 sp=0xc00006ec78 pc=0x45ff09
internal/poll.(*pollDesc).wait(0xc00034eb80?, 0xc0003bd000?, 0x0)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00006ecc0 sp=0xc00006ec98 pc=0x4b2f52
internal/poll.(*pollDesc).waitRead(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00034eb80, {0xc0003bd000, 0x5, 0x5000})
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc00006ed58 sp=0xc00006ecc0 pc=0x4b3cf9
net.(*netFD).Read(0xc00034eb80, {0xc0003bd000?, 0xc00006ee08?, 0xc00006ede0?})
/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29 fp=0xc00006eda0 sp=0xc00006ed58 pc=0x4eb489
net.(*conn).Read(0xc000012428, {0xc0003bd000?, 0x6d4b20?, 0xc0003d0150?})
/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45 fp=0xc00006ede8 sp=0xc00006eda0 pc=0x4f6b45
net.(*TCPConn).Read(0xc00034ed00?, {0xc0003bd000?, 0xc00006ee50?, 0x40adf6?})
:1 +0x29 fp=0xc00006ee18 sp=0xc00006ede8 pc=0x504009
io.ReadAtLeast({0x7b5820, 0xc000012428}, {0xc0003bd000, 0x5, 0x5000}, 0x5)
/opt/hostedtoolcache/go/1.20.4/x64/src/io/io.go:332 +0x9a fp=0xc00006ee60 sp=0xc00006ee18 pc=0x46d41a
io.ReadFull(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/io/io.go:351
github.com/cbeuw/Cloak/internal/common.(*TLSConn).Read(0xc00037ec00, {0xc0003bd000, 0x5000, 0x5000})
/home/runner/work/Cloak/Cloak/internal/common/tls.go:82 +0x72 fp=0xc00006eeb8 sp=0xc00006ee60 pc=0x634cb2
github.com/cbeuw/Cloak/internal/client.(*DirectTLS).Read(0xc000357680?, {0xc0003bd000?, 0xa1?, 0x5000?})
:1 +0x2a fp=0xc00006eee8 sp=0xc00006eeb8 pc=0x6bd5aa
github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).deplex(0xc00034ec80, {0x7f9beda3a218?, 0xc000011b30})
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:161 +0xff fp=0xc00006efb8 sp=0xc00006eee8 pc=0x64df7f
github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).addConn.func1()
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:65 +0x2e fp=0xc00006efe0 sp=0xc00006efb8 pc=0x64d74e
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00006efe8 sp=0xc00006efe0 pc=0x4650a1
created by github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).addConn
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:65 +0xd5

goroutine 96 [IO wait, 32 minutes]:
runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0xb?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc00003ac40 sp=0xc00003ac20 pc=0x436b96
runtime.netpollblock(0x49cca5?, 0x40404f?, 0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:527 +0xf7 fp=0xc00003ac78 sp=0xc00003ac40 pc=0x42f597
internal/poll.runtime_pollWait(0x7f9beda2fbb8, 0x72)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89 fp=0xc00003ac98 sp=0xc00003ac78 pc=0x45ff09
internal/poll.(*pollDesc).wait(0xc00034ec00?, 0xc0003ea000?, 0x0)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00003acc0 sp=0xc00003ac98 pc=0x4b2f52
internal/poll.(*pollDesc).waitRead(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00034ec00, {0xc0003ea000, 0x5, 0x5000})
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc00003ad58 sp=0xc00003acc0 pc=0x4b3cf9
net.(*netFD).Read(0xc00034ec00, {0xc0003ea000?, 0x7f9beda36598?, 0x0?})
/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29 fp=0xc00003ada0 sp=0xc00003ad58 pc=0x4eb489
net.(*conn).Read(0xc000012418, {0xc0003ea000?, 0x4205e5?, 0x9cdcc0?})
/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45 fp=0xc00003ade8 sp=0xc00003ada0 pc=0x4f6b45
net.(*TCPConn).Read(0xc00001a87d?, {0xc0003ea000?, 0xc00003ae50?, 0x40adf6?})
:1 +0x29 fp=0xc00003ae18 sp=0xc00003ade8 pc=0x504009
io.ReadAtLeast({0x7b5820, 0xc000012418}, {0xc0003ea000, 0x5, 0x5000}, 0x5)
/opt/hostedtoolcache/go/1.20.4/x64/src/io/io.go:332 +0x9a fp=0xc00003ae60 sp=0xc00003ae18 pc=0x46d41a
io.ReadFull(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/io/io.go:351
github.com/cbeuw/Cloak/internal/common.(*TLSConn).Read(0xc00037eb40, {0xc0003ea000, 0x5000, 0x5000})
/home/runner/work/Cloak/Cloak/internal/common/tls.go:82 +0x72 fp=0xc00003aeb8 sp=0xc00003ae60 pc=0x634cb2
github.com/cbeuw/Cloak/internal/client.(*DirectTLS).Read(0xc000356a00?, {0xc0003ea000?, 0x20?, 0x5000?})
:1 +0x2a fp=0xc00003aee8 sp=0xc00003aeb8 pc=0x6bd5aa
github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).deplex(0xc00034ec80, {0x7f9beda3a218?, 0xc000011ae8})
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:161 +0xff fp=0xc00003afb8 sp=0xc00003aee8 pc=0x64df7f
github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).addConn.func1()
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:65 +0x2e fp=0xc00003afe0 sp=0xc00003afb8 pc=0x64d74e
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00003afe8 sp=0xc00003afe0 pc=0x4650a1
created by github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).addConn
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:65 +0xd5

goroutine 99 [runnable]:
runtime.gopark(0xc00034eb80?, 0xb?, 0x0?, 0x0?, 0x7?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000071bc8 sp=0xc000071ba8 pc=0x436b96
runtime.netpollblock(0x49cca5?, 0x40404f?, 0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:527 +0xf7 fp=0xc000071c00 sp=0xc000071bc8 pc=0x42f597
internal/poll.runtime_pollWait(0x7f9beda2fe88, 0x72)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89 fp=0xc000071c20 sp=0xc000071c00 pc=0x45ff09
internal/poll.(*pollDesc).wait(0xc00034e900?, 0xc00039880e?, 0x0)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000071c48 sp=0xc000071c20 pc=0x4b2f52
internal/poll.(*pollDesc).waitRead(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00034e900, {0xc00039880e, 0x3ff3, 0x4003})
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc000071ce0 sp=0xc000071c48 pc=0x4b3cf9
net.(*netFD).Read(0xc00034e900, {0xc00039880e?, 0xc00039880e?, 0xc000071d90?})
/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29 fp=0xc000071d28 sp=0xc000071ce0 pc=0x4eb489
net.(*conn).Read(0xc0000123d0, {0xc00039880e?, 0x0?, 0xc000011cc8?})
/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45 fp=0xc000071d70 sp=0xc000071d28 pc=0x4f6b45
net.(*TCPConn).Read(0xc000357730?, {0xc00039880e?, 0xc000011cc8?, 0x4011?})
:1 +0x29 fp=0xc000071da0 sp=0xc000071d70 pc=0x504009
github.com/cbeuw/Cloak/internal/multiplex.(*Stream).ReadFrom(0xc00034ed00, {0x7b5820, 0xc0000123d0})
/home/runner/work/Cloak/Cloak/internal/multiplex/stream.go:174 +0x18a fp=0xc000071e28 sp=0xc000071da0 pc=0x64b6aa
github.com/cbeuw/Cloak/internal/common.Copy({0x7b94c0, 0xc00034ed00}, {0x7b95c8, 0xc0000123d0})
/home/runner/work/Cloak/Cloak/internal/common/copy.go:50 +0x164 fp=0xc000071ef0 sp=0xc000071e28 pc=0x6341c4
github.com/cbeuw/Cloak/internal/client.RouteTCP.func2(0x0?, {0x7b95c8?, 0xc0000123d0}, 0x1c?)
/home/runner/work/Cloak/Cloak/internal/client/piper.go:186 +0x33b fp=0xc000071fb0 sp=0xc000071ef0 pc=0x6b5edb
github.com/cbeuw/Cloak/internal/client.RouteTCP.func3()
/home/runner/work/Cloak/Cloak/internal/client/piper.go:189 +0x36 fp=0xc000071fe0 sp=0xc000071fb0 pc=0x6b5b76
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc000071fe8 sp=0xc000071fe0 pc=0x4650a1
created by github.com/cbeuw/Cloak/internal/client.RouteTCP
/home/runner/work/Cloak/Cloak/internal/client/piper.go:147 +0x408

goroutine 102 [GC worker (idle), 4 minutes]:
runtime.gopark(0xed82222352c6?, 0xae36ef778a97a7b7?, 0x75?, 0x8b?, 0xa914617c5e74a53b?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc000034750 sp=0xc000034730 pc=0x436b96
runtime.gcBgMarkWorker()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1275 +0xf1 fp=0xc0000347e0 sp=0xc000034750 pc=0x41a231
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc0000347e8 sp=0xc0000347e0 pc=0x4650a1
created by runtime.gcBgMarkStartWorkers
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/mgc.go:1199 +0x25

goroutine 98 [IO wait, 32 minutes]:
runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0x8?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc00003a440 sp=0xc00003a420 pc=0x436b96
runtime.netpollblock(0x49cca5?, 0x40404f?, 0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:527 +0xf7 fp=0xc00003a478 sp=0xc00003a440 pc=0x42f597
internal/poll.runtime_pollWait(0x7f9beda2fca8, 0x72)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89 fp=0xc00003a498 sp=0xc00003a478 pc=0x45ff09
internal/poll.(*pollDesc).wait(0xc00034ea80?, 0xc0003f4000?, 0x0)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00003a4c0 sp=0xc00003a498 pc=0x4b2f52
internal/poll.(*pollDesc).waitRead(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00034ea80, {0xc0003f4000, 0x5, 0x5000})
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc00003a558 sp=0xc00003a4c0 pc=0x4b3cf9
net.(*netFD).Read(0xc00034ea80, {0xc0003f4000?, 0x7f9beda364f8?, 0x0?})
/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29 fp=0xc00003a5a0 sp=0xc00003a558 pc=0x4eb489
net.(*conn).Read(0xc000012438, {0xc0003f4000?, 0x4205e5?, 0x9cdcc0?})
/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45 fp=0xc00003a5e8 sp=0xc00003a5a0 pc=0x4f6b45
net.(*TCPConn).Read(0xc00001a87d?, {0xc0003f4000?, 0xc00003a650?, 0x40adf6?})
:1 +0x29 fp=0xc00003a618 sp=0xc00003a5e8 pc=0x504009
io.ReadAtLeast({0x7b5820, 0xc000012438}, {0xc0003f4000, 0x5, 0x5000}, 0x5)
/opt/hostedtoolcache/go/1.20.4/x64/src/io/io.go:332 +0x9a fp=0xc00003a660 sp=0xc00003a618 pc=0x46d41a
io.ReadFull(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/io/io.go:351
github.com/cbeuw/Cloak/internal/common.(*TLSConn).Read(0xc00037ecc0, {0xc0003f4000, 0x5000, 0x5000})
/home/runner/work/Cloak/Cloak/internal/common/tls.go:82 +0x72 fp=0xc00003a6b8 sp=0xc00003a660 pc=0x634cb2
github.com/cbeuw/Cloak/internal/client.(*DirectTLS).Read(0x1000000006b5d1d?, {0xc0003f4000?, 0x0?, 0xc000369800?})
:1 +0x2a fp=0xc00003a6e8 sp=0xc00003a6b8 pc=0x6bd5aa
github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).deplex(0xc00034ec80, {0x7f9beda3a218?, 0xc000011b78})
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:161 +0xff fp=0xc00003a7b8 sp=0xc00003a6e8 pc=0x64df7f
github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).addConn.func1()
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:65 +0x2e fp=0xc00003a7e0 sp=0xc00003a7b8 pc=0x64d74e
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00003a7e8 sp=0xc00003a7e0 pc=0x4650a1
created by github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).addConn
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:65 +0xd5

goroutine 100 [sync.Cond.Wait]:
runtime.gopark(0xc0000548a0?, 0xc00006ad40?, 0x49?, 0x42?, 0xc0000123d0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc00006ad08 sp=0xc00006ace8 pc=0x436b96
runtime.goparkunlock(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:387
sync.runtime_notifyListWait(0xc00037ee90, 0x43cf)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/sema.go:527 +0x14c fp=0xc00006ad50 sp=0xc00006ad08 pc=0x461aac
sync.(*Cond).Wait(0xc0003d0060?)
/opt/hostedtoolcache/go/1.20.4/x64/src/sync/cond.go:70 +0x8c fp=0xc00006ad88 sp=0xc00006ad50 pc=0x46a18c
github.com/cbeuw/Cloak/internal/multiplex.(*streamBufferedPipe).WriteTo(0xc00037eec0, {0x7b5840, 0xc0000123d0})
/home/runner/work/Cloak/Cloak/internal/multiplex/streamBufferedPipe.go:93 +0x216 fp=0xc00006ae08 sp=0xc00006ad88 pc=0x64cc76
github.com/cbeuw/Cloak/internal/multiplex.(*streamBuffer).WriteTo(0x73f720?, {0x7b5840?, 0xc0000123d0?})
/home/runner/work/Cloak/Cloak/internal/multiplex/streamBuffer.go:106 +0x27 fp=0xc00006ae30 sp=0xc00006ae08 pc=0x64c447
github.com/cbeuw/Cloak/internal/multiplex.(*Stream).WriteTo(0xc00034ed00, {0x7b5840?, 0xc0000123d0?})
/home/runner/work/Cloak/Cloak/internal/multiplex/stream.go:102 +0x4f fp=0xc00006aec0 sp=0xc00006ae30 pc=0x64ae6f
github.com/cbeuw/Cloak/internal/common.Copy({0x7b95c8, 0xc0000123d0}, {0x7b94c0, 0xc00034ed00})
/home/runner/work/Cloak/Cloak/internal/common/copy.go:46 +0x1e6 fp=0xc00006af88 sp=0xc00006aec0 pc=0x634246
github.com/cbeuw/Cloak/internal/client.RouteTCP.func2.1()
/home/runner/work/Cloak/Cloak/internal/client/piper.go:182 +0x2c fp=0xc00006afe0 sp=0xc00006af88 pc=0x6b5fac
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00006afe8 sp=0xc00006afe0 pc=0x4650a1
created by github.com/cbeuw/Cloak/internal/client.RouteTCP.func2
/home/runner/work/Cloak/Cloak/internal/client/piper.go:181 +0x31d

goroutine 97 [IO wait, 32 minutes]:
runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0x9?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/proc.go:381 +0xd6 fp=0xc00003bc40 sp=0xc00003bc20 pc=0x436b96
runtime.netpollblock(0x49cca5?, 0x40404f?, 0x0?)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:527 +0xf7 fp=0xc00003bc78 sp=0xc00003bc40 pc=0x42f597
internal/poll.runtime_pollWait(0x7f9beda2fd98, 0x72)
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/netpoll.go:306 +0x89 fp=0xc00003bc98 sp=0xc00003bc78 pc=0x45ff09
internal/poll.(*pollDesc).wait(0xc00034eb00?, 0xc0003ef000?, 0x0)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00003bcc0 sp=0xc00003bc98 pc=0x4b2f52
internal/poll.(*pollDesc).waitRead(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00034eb00, {0xc0003ef000, 0x5, 0x5000})
/opt/hostedtoolcache/go/1.20.4/x64/src/internal/poll/fd_unix.go:167 +0x299 fp=0xc00003bd58 sp=0xc00003bcc0 pc=0x4b3cf9
net.(*netFD).Read(0xc00034eb00, {0xc0003ef000?, 0x0?, 0x0?})
/opt/hostedtoolcache/go/1.20.4/x64/src/net/fd_posix.go:55 +0x29 fp=0xc00003bda0 sp=0xc00003bd58 pc=0x4eb489
net.(*conn).Read(0xc000012448, {0xc0003ef000?, 0x7b7de8?, 0xc0000180e0?})
/opt/hostedtoolcache/go/1.20.4/x64/src/net/net.go:183 +0x45 fp=0xc00003bde8 sp=0xc00003bda0 pc=0x4f6b45
net.(*TCPConn).Read(0xc00001a870?, {0xc0003ef000?, 0xc00003be50?, 0x40adf6?})
:1 +0x29 fp=0xc00003be18 sp=0xc00003bde8 pc=0x504009
io.ReadAtLeast({0x7b5820, 0xc000012448}, {0xc0003ef000, 0x5, 0x5000}, 0x5)
/opt/hostedtoolcache/go/1.20.4/x64/src/io/io.go:332 +0x9a fp=0xc00003be60 sp=0xc00003be18 pc=0x46d41a
io.ReadFull(...)
/opt/hostedtoolcache/go/1.20.4/x64/src/io/io.go:351
github.com/cbeuw/Cloak/internal/common.(*TLSConn).Read(0xc00037ed80, {0xc0003ef000, 0x5000, 0x5000})
/home/runner/work/Cloak/Cloak/internal/common/tls.go:82 +0x72 fp=0xc00003beb8 sp=0xc00003be60 pc=0x634cb2
github.com/cbeuw/Cloak/internal/client.(*DirectTLS).Read(0xc000356a00?, {0xc0003ef000?, 0x20?, 0x5000?})
:1 +0x2a fp=0xc00003bee8 sp=0xc00003beb8 pc=0x6bd5aa
github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).deplex(0xc00034ec80, {0x7f9beda3a218?, 0xc000011bc0})
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:161 +0xff fp=0xc00003bfb8 sp=0xc00003bee8 pc=0x64df7f
github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).addConn.func1()
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:65 +0x2e fp=0xc00003bfe0 sp=0xc00003bfb8 pc=0x64d74e
runtime.goexit()
/opt/hostedtoolcache/go/1.20.4/x64/src/runtime/asm_amd64.s:1598 +0x1 fp=0xc00003bfe8 sp=0xc00003bfe0 pc=0x4650a1
created by github.com/cbeuw/Cloak/internal/multiplex.(*switchboard).addConn
/home/runner/work/Cloak/Cloak/internal/multiplex/switchboard.go:65 +0xd5

The config is :
{
"Transport": "direct",
"ProxyMethod": "socks5tcp",
"EncryptionMethod": "aes-128-gcm",
"UID": "omitted",
"PublicKey": "omitted ",
"ServerName": "console.aws.amazon.com",
"CDNOriginHost": "console.aws.amazon.com",
"NumConn": 4,
"BrowserSig": "chrome",
"StreamTimeout": 300,
"KeepAlive": 299,
"LoopbackTcpSendBuffer": 5120,
"LoopbackTcpReceiveBuffer": 5120
}

The reason for using keepalives is that I have found this value of Keepalive to be substantially protective against stalls (related issue) (1 severe stall per multiple hours versus one per hour or more)

Hope this will help improving Cloak :-)

@notsure2
Copy link
Contributor

That's coming from deep in golang runtime, not from cloak itself, it might a bug there (very low probability), or you may have some kind hw issue (maybe too aggressive overclock or failing ram ?)

@LindaFerum
Copy link
Author

Well, I can't really positively rule it out since it's a small "industrial" unit with SSD being the only user replaceable part

When the job is done I may run some stress tests on it later but the only thing it is doing is being a gateway and cloakifying a particular TCP connection so it isn't under huge big load.

@LindaFerum
Copy link
Author

@notsure2 for what it's worth I've been able to produce very similar crash behavior on a rather well maintained laptop with fresh Qubes install by simply letting TCP shadowsocks "hard stall" and then not touching it for like two hours or so

Thus unlikely to be hardware fault (well, two different machines may have both have subtle hardware problems that manifest similarly, but unlikely)

@notsure2
Copy link
Contributor

checkdead is the golang deadlock detector. it crashes the process if it finds that no goroutines are running. It could be a real deadlock in cloak or an incorrect deadlock detection due to no data passing. I'm not sure.

@LindaFerum
Copy link
Author

Well one thing I am certain of is that the software on the user side of Cloak client that crashed was desperately trying to send some data :-)
Plus the keepalive is on, so it SHOULD have been sending something at least

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants