Three+ built-in-go setup doc?

[alecsavvy]

Hello!

I'm experimenting a bit with cometbft, in particular the built-in-go example here: https://github.com/cometbft/cometbft/blob/main/docs/tutorials/go-built-in.md. However, when I try to expand this to a three node (or more) cluster I regularly get consensus failures with my nodes. I've setup config for multiple nodes using the cometbft testnet command and modified the ports so they don't conflict. I'm running these with go run ., not docker.

Here's the error I run into when trying to add to the KV store with three nodes.

{"time":"2024-04-25T10:51:34.560628-06:00","level":"ERROR","msg":"CONSENSUS FAILURE!!!","module":"consensus","err":"failed to apply block; error commit failed for application: Writes are blocked, possibly due to DropAll or Close","stack":"goroutine 153 [running]:\nruntime/debug.Stack()\n\t/Users/alecsavoy/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.2.darwin-arm64/src/runtime/debug/stack.go:24 +0x64\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\t/Users/alecsavoy/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:801 +0x44\npanic({0x1059ebee0?, 0x1400004e490?})\n\t/Users/alecsavoy/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.22.2.darwin-arm64/src/runtime/panic.go:770 +0x124\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0x1400041b888, 0x15)\n\t/Users/alecsavoy/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:1780 +0xad0\ngithub.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0x1400041b888, 0x15)\n\t/Users/alecsavoy/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:1682 +0x26c\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit.func1()\n\t/Users/alecsavoy/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:1617 +0x8c\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit(0x1400041b888, 0x15, 0x0)\n\t/Users/alecsavoy/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:1655 +0xac4\ngithub.com/cometbft/cometbft/consensus.(*State).addVote(0x1400041b888, 0x1400ab74270, {0x14006f96150, 0x28})\n\t/Users/alecsavoy/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:2334 +0x1858\ngithub.com/cometbft/cometbft/consensus.(*State).tryAddVote(0x1400041b888, 0x1400ab74270, {0x14006f96150?, 0x1054dfb14?})\n\t/Users/alecsavoy/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:2066 +0x28\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg(0x1400041b888, {{0x105bc3d20, 0x14000486150}, {0x14006f96150, 0x28}})\n\t/Users/alecsavoy/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:929 +0x300\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine(0x1400041b888, 0x0)\n\t/Users/alecsavoy/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:836 +0x2c4\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutine 103\n\t/Users/alecsavoy/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:398 +0xf0\n"}

The nodes freeze after this (as the error says) but I'm not sure why. I'm using the KV store in the example so I'm assuming it should be just a pass through. Any thoughts?