fatal error: mSpanList.insert

[yunpliu]

Go version

go1.18.10

Output of go env in your module/workspace:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/.gvm/pkgsets/go1.18.10/global/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/.gvm/pkgsets/go1.18.10/global"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/root/.gvm/gos/go1.18.10"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/root/.gvm/gos/go1.18.10/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.10"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/jenkins/workspace/uat-server-devops-yl-devops-eventengine/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build545321703=/tmp/go-build -gno-record-gcc-switches"

What did you do?

It happened occasionally, I can't reproduced it stably.

What did you see happen?

runtime: failed mSpanList.insert 0x7f389c85b908 0x9b05688c510e527f 0x5be0cd191f83d9ab 0x0
fatal error: mSpanList.insert

runtime stack:
runtime.throw({0xe37346?, 0x0?})
/root/.gvm/gos/go1.18.10/src/runtime/panic.go:992 +0x71
runtime.(*mSpanList).insert(0xc000f0fde8?, 0x7f389c85b908)
/root/.gvm/gos/go1.18.10/src/runtime/mheap.go:1622 +0xe5
runtime.getempty()
/root/.gvm/gos/go1.18.10/src/runtime/mgcwork.go:382 +0x105
runtime.(*gcWork).put(0xc000081238, 0xc00104dba0)
/root/.gvm/gos/go1.18.10/src/runtime/mgcwork.go:128 +0x67
runtime.greyobject(0xc00104dba0, 0xc000f0fed0?, 0x432705?, 0xc001606948?, 0xc000081270?, 0x428011?)
/root/.gvm/gos/go1.18.10/src/runtime/mgcmark.go:1486 +0x19d
runtime.scanobject(0xc000081238?, 0xc000081238)
/root/.gvm/gos/go1.18.10/src/runtime/mgcmark.go:1317 +0x1b2
runtime.gcDrain(0xc000081238, 0x3)
/root/.gvm/gos/go1.18.10/src/runtime/mgcmark.go:1081 +0x1d4
runtime.gcBgMarkWorker.func2()
/root/.gvm/gos/go1.18.10/src/runtime/mgc.go:1276 +0xa5
runtime.systemstack()
/root/.gvm/gos/go1.18.10/src/runtime/asm_amd64.s:469 +0x49

What did you expect to see?

no fatal error and process panic

[Jorropo]

This is an old version of go that is unsupported.
Does this happen with a version of go that is supported ?

• https://endoflife.date/go
• https://go.dev/wiki/Go-Release-Cycle#release-maintenance

[yunpliu]

Not yet, I am trying to reproduce it with go1.22.

[yunpliu]

@Jorropo After I upgrade go version to 1.22.0, I didn't reproduce the error above, But I encounter some new case as below, It can't be reproduced stably.

fatal error: runtime: sudog with non-nil next

goroutine 6095085 gp=0xc000d0c8c0 m=7 mp=0xc000680008 [running]:
runtime.throw({0xe4c8e1?, 0xc00199d708?})
        /root/.gvm/gos/go1.22.0/src/runtime/panic.go:1023 +0x5c fp=0xc00199d6c8 sp=0xc00199d698 pc=0x43cffc
runtime.releaseSudog(0x15d1520?)
        /root/.gvm/gos/go1.22.0/src/runtime/proc.go:464 +0x2fb fp=0xc00199d738 sp=0xc00199d6c8 pc=0x44053b
runtime.semacquire1(0xc00015a0cc, 0x0, 0x1, 0x0, 0x12)
        /root/.gvm/gos/go1.22.0/src/runtime/sema.go:168 +0x2cf fp=0xc00199d7a0 sp=0xc00199d738 pc=0x4522af
internal/poll.runtime_Semacquire(0x0?)
        /root/.gvm/gos/go1.22.0/src/runtime/sema.go:67 +0x25 fp=0xc00199d7d8 sp=0xc00199d7a0 pc=0x46e8a5
internal/poll.(*fdMutex).rwlock(0xc00015a0c0, 0x80?)
        /root/.gvm/gos/go1.22.0/src/internal/poll/fd_mutex.go:154 +0xc5 fp=0xc00199d818 sp=0xc00199d7d8 pc=0x4ddae5
internal/poll.(*FD).writeLock(...)
        /root/.gvm/gos/go1.22.0/src/internal/poll/fd_mutex.go:239
internal/poll.(*FD).Write(0xc00015a0c0, {0xc0005c20c0, 0x77, 0xc0})
        /root/.gvm/gos/go1.22.0/src/internal/poll/fd_unix.go:367 +0x65 fp=0xc00199d8c8 sp=0xc00199d818 pc=0x4e1125
os.(*File).write(...)
        /root/.gvm/gos/go1.22.0/src/os/file_posix.go:46
os.(*File).Write(0xc00013a058, {0xc0005c20c0?, 0x77, 0xc00199db88?})
        /root/.gvm/gos/go1.22.0/src/os/file.go:189 +0x51 fp=0xc00199d928 sp=0xc00199d8c8 pc=0x4ea191
fmt.Fprintf({0xf70b60, 0xc00013a058}, {0xe499c7, 0x1d}, {0xc0008d1b88, 0x3, 0x3})
        /root/.gvm/gos/go1.22.0/src/fmt/print.go:225 +0x97 fp=0xc00199d988 sp=0xc00199d928 pc=0x4f6077
fmt.Printf(...)
        /root/.gvm/gos/go1.22.0/src/fmt/print.go:233
eventengine/pkg/expr.SilentRuleCheck({_, {_, _}, {_, _}, _, {_, _, _}}, {{0xc0010620c0, ...}, ...})
        /home/jenkins/workspace/uat-server-devops-yl-devops-eventengine/pkg/expr/expr.go:53 +0x1ef fp=0xc00199dbc8 sp=0xc00199d988 pc=0xa6428f
eventengine/consumer.SlientRulesProcess.func1({{0xc0010620c0, 0x16}, {0xc000c20128, 0x6}, {0xc000c20140, 0x3}, {0x0, 0x0}, {0xc0006ba0d0, 0xc6}, ...}, ...)
        /home/jenkins/workspace/uat-server-devops-yl-devops-eventengine/consumer/consumer.go:243 +0x105 fp=0xc00199dde8 sp=0xc00199dbc8 pc=0xb35605
eventengine/consumer.SlientRulesProcess.gowrap1()
        /home/jenkins/workspace/uat-server-devops-yl-devops-eventengine/consumer/consumer.go:249 +0xa3 fp=0xc00199dfe0 sp=0xc00199dde8 pc=0xb354c3
runtime.goexit({})
        /root/.gvm/gos/go1.22.0/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00199dfe8 sp=0xc00199dfe0 pc=0x4721a1
created by eventengine/consumer.SlientRulesProcess in goroutine 6088462
        /home/jenkins/workspace/uat-server-devops-yl-devops-eventengine/consumer/consumer.go:241 +0x58c