Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug(engine): goroutine leaks #170

Open
9 tasks
steebchen opened this issue Feb 16, 2024 · 0 comments
Open
9 tasks

bug(engine): goroutine leaks #170

steebchen opened this issue Feb 16, 2024 · 0 comments

Comments

@steebchen
Copy link
Member

steebchen commented Feb 16, 2024 •
edited

List of goroutine leaks which are currently ignored, but should be investigate if they could become a problem.

Related code:

hatchet/examples/loadtest/cli/cli_e2e_test.go

Lines 54 to 57 in 00111d8

goleak.IgnoreTopFunction("go.opencensus.io/stats/view.(*worker).start"),
goleak.IgnoreTopFunction("google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run"),
goleak.IgnoreTopFunction("internal/poll.runtime_pollWait"),
goleak.IgnoreTopFunction("google.golang.org/grpc/internal/transport.(*controlBuffer).get"),

Goroutine leaks:

  • go.opencensus.io/stats/view.(*worker).start

    [Goroutine 6 in state select, with go.opencensus.io/stats/view.(*worker).start on top of the stack:
go.opencensus.io/stats/view.(*worker).start(0xc0002feb80)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x128
created by go.opencensus.io/stats/view.init.0 in goroutine 1
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0xf4

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 23 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373500, {0x102fde440, 0xc00004aaf0})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x1f8

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 24 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373530, {0x102fde440, 0xc00004ab40})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x1f8

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 25 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000373560, {0x102fde440, 0xc00004ab90})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 54
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x1f8
 Goroutine 40 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
internal/poll.runtime_pollWait(0x14d67ce38, 0x72)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc00052c0a0, 0xc0000dd200?, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00052c080, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_unix.go:164 +0x2e0
net.(*netFD).Read(0xc00052c080, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/fd_posix.go:55 +0x48
net.(*conn).Read(0xc00051e018, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/net.go:179 +0x8c
crypto/tls.(*atLeastReader).Read(0xc00069a870, {0xc0000dd200, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:805 +0x7c
bytes.(*Buffer).ReadFrom(0xc00060a2a8, {0x102fd69b8, 0xc00069a870})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bytes/buffer.go:211 +0xf4
crypto/tls.(*Conn).readFromUntil(0xc00060a000, {0x14d67cf30?, 0xc00051e018}, 0x5)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:827 +0x18c
crypto/tls.(*Conn).readRecordOrCCS(0xc00060a000, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:625 +0x480
crypto/tls.(*Conn).readRecord(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc00060a000, {0xc0005e2000, 0x8000, 0x50000c0003bfb38?})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:1369 +0x1c4
bufio.(*Reader).Read(0xc000594d80, {0xc000596200, 0x9, 0x9})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bufio/bufio.go:244 +0x390
io.ReadAtLeast({0x102fd5e58, 0xc000594d80}, {0xc000596200, 0x9, 0x9}, 0x9)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:335 +0xcc
io.ReadFull(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc000596200, 0x9, 0x9}, {0x102fd5e58, 0xc000594d80})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:237 +0x68
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005961c0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:498 +0xbc
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00057c240, 0xc00060a000?)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1587 +0x1e8
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 26
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:398 +0x2234

  • google.golang.org/grpc/internal/transport.(*controlBuffer).get

     Goroutine 41 in state select, with google.golang.org/grpc/internal/transport.(*controlBuffer).get on top of the stack:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0004e4460, 0x1)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:418 +0x120
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0000e77a0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:552 +0xf0
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:452 +0x114
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 26
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:450 +0x288c

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 55 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b920, {0x102fde440, 0xc00004ac30})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x1f8

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 56 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b950, {0x102fde440, 0xc00004ad70})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x1f8

  • google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run

     Goroutine 57 in state select, with google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run on top of the stack:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc00012b980, {0x102fde440, 0xc00004adc0})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:76 +0x150
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 10
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/grpcsync/callback_serializer.go:52 +0x1f8
 Goroutine 61 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:
internal/poll.runtime_pollWait(0x14d67cd40, 0x72)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0xc00021dda0, 0xc0000ddb00?, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:84 +0xb8
internal/poll.(*pollDesc).waitRead(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00021dd80, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/internal/poll/fd_unix.go:164 +0x2e0
net.(*netFD).Read(0xc00021dd80, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/fd_posix.go:55 +0x48
net.(*conn).Read(0xc00051e0d0, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/net/net.go:179 +0x8c
crypto/tls.(*atLeastReader).Read(0xc000155d10, {0xc0000ddb00, 0x900, 0x900})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:805 +0x7c
bytes.(*Buffer).ReadFrom(0xc00060b0a8, {0x102fd69b8, 0xc000155d10})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bytes/buffer.go:211 +0xf4
crypto/tls.(*Conn).readFromUntil(0xc00060ae00, {0x14d67cf30?, 0xc00051e0d0}, 0x5)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:827 +0x18c
crypto/tls.(*Conn).readRecordOrCCS(0xc00060ae00, 0x0)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:625 +0x480
crypto/tls.(*Conn).readRecord(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:587
crypto/tls.(*Conn).Read(0xc00060ae00, {0xc000440000, 0x8000, 0x50000c0005258b8?})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/crypto/tls/conn.go:1369 +0x1c4
bufio.(*Reader).Read(0xc00018f140, {0xc0000d63c0, 0x9, 0x9})
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/bufio/bufio.go:244 +0x390
io.ReadAtLeast({0x102fd5e58, 0xc00018f140}, {0xc0000d63c0, 0x9, 0x9}, 0x9)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:335 +0xcc
io.ReadFull(...)
  /Users/steebchen/.asdf/installs/golang/1.21.5/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0000d63c0, 0x9, 0x9}, {0x102fd5e58, 0xc00018f140})
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:237 +0x68
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0000d6380)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/golang.org/x/net@v0.20.0/http2/frame.go:498 +0xbc
google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00057cd80, 0xc00060ae00?)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:1587 +0x1e8
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 14
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:398 +0x2234

  • google.golang.org/grpc/internal/transport.(*controlBuffer).get

     Goroutine 62 in state select, with google.golang.org/grpc/internal/transport.(*controlBuffer).get on top of the stack:
google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00004b540, 0x1)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:418 +0x120
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000520d20)
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/controlbuf.go:552 +0xf0
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:452 +0x114
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 14
  /Users/steebchen/.asdf/installs/golang/1.21.5/packages/pkg/mod/google.golang.org/grpc@v1.60.1/internal/transport/http2_client.go:450 +0x288c
]
@steebchen steebchen changed the title Goroutine leaks bug(engine): goroutine leaks Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@steebchen