[testnet shard0] - node started from genesis have stopped validation on the 20627530 due to invalid merkle root

[mur-me]

Describe the bug
New node in the testent shard0 have stopped validation on the 20627530 block due to invalid merkle root

To Reproduce

1. Create new node in the testnet shard0 from genesis
2. Wait till the 20627530 block

Expected behavior
Node should proceed with block validation and catch up with network

Logs:

{"level":"error","caller":"/home/uladzislau/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:2070","time":"2024-03-22T17:00:53.947922101+01:00","message":"
########## BAD BLOCK #########
Chain config: {ChainID: 2 EthCompatibleChainID: 1666700000 EIP155: 0 Cr
ossTx: 0 Staking: 2 CrossLink: 2 ReceiptLog: 0 SHA3Epoch: 0 StakingPrecompileEpoch: 2 ChainIdFixEpoch: 0 CrossShardXferPrecompileEpoch: 2}

Number: 20627531
Epoch: 2519
NumTxn: 0
NumStkTxn: 0
Hash: 0x0bffb55061bee4e44edf48617f6d0b9bd53f26040792687a0eeed940ecb7c5bf
Error: invalid merkle root (remote: 72f5b545336ce699181269ee1172b92b818aaf1b3bcd973a55eee710397aeb44 local: 74a1097177791020ff54158a41c3f794cd5ef8d22d6e2fc40dbf9cccaa84ac27, rlp dump f9087387486d6e79546764827633f90865a019760fd428377ef70a8bdf1652c30232bdb62f85cb12d018b65
026a43979138494350762b374e56690a5055d936b28c2244a92ac28a072f5b545336ce699181269ee1172b92b818aaf1b3bcd973a55eee710397aeb44a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421a056e81f171bcc55a6ff
8345e692c0f86e5b48e01b996cadc001622fb5e363b421a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421b9010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000000084013ac04b8401c9c3808084657d95f180a0000000000000000000000000000000000000000000000000000000000000000084013acf3a8209d780b8607b1f7a0c81e8e2bc83480e5fcaf5d361492708fcfafc72399f96396a059c9992531
f574b4edf67b619139ca6b81c940bc4f8b9806dfeed3ac5e456970355e877c40a9d0d12764fbd6c651549809d9212cfc926cf2708bea42c78e33b80c4a28882fe03b8809e0fed56f2455dc7bb5e9403f5f0fcd08a3ec13776acdc1cb8eea2e6e39caaeef2d3b3f4d8a2b70aff7ed1646e54e31ac9f7e7503d8ff5216cebc2aba1f963920fa6b81862
45b4680a1580a65e78c612e2f2c0ed9557b998a26a37f94d14f865f1024dc859ae4990fc7fc2d8df9d8a7690ccb67c3c797906d6d1821a8e977d8c80b9055ff9055c8209d8f90556f9030480f90300f84794261fa45c6a09cd3faa277d829e91d9473973357cb0e7f54994bc5c02edeeb178ce2d34db276a893bab5c59ac3d7eb9f077c893f9e3117
1de6236ba0e21be415d8631e45b91c0f847940f25138bbbbff872e662c364ed0ff7f81640ab1ab099d0835797ca0683fb7b1d14a882879652ddcdcfe0d52385ffddf8012ee804d92e5c05a56c9d7fc663678e36a158a28cc0f8479468771f0430433219248c8777d82a9b6a8efc2d1bb08a211eb5e9334341fd2498fb5d6b922b4a0984d6a4ea0b56
31c1904de5fe21fd6889c9c032d862546ca50a5c41294b0cc0f847944092ac6d9a7f87aacd7787b5c1c4541d9bf2cd5cb01833721b78797a16b1987734d05b08b9444e24075fda50ff2acc7b8a6d8e0aef0829bcb11e3b9df7466cf8a39e4e4101c0f8479450be91354b22668c22f0f8f6ca78db9fb38ae018b08c95e04a4826d4d80ef16183f13aa
5d14eb3c96d2755407e15c440bb4edd6e4636a82e47975385c6223ba24759561103c0f84794234f723b141dfb0d47eadccbda96e838a699dfaab0a2b1b534c94bf19a92551f1d32d62b802be6996458f65b0baeb081c9972b34d72310e675fc7797b9c860d8cc223cee8fc0f847945f729d82a59b29678ec355a55330f38f7d4dd59bb0f248bd21d6
7f0b2cd0dd2c06446c557fc35737873857c000698ae391b607ca8ed8df00a79d9dcace1b0ce05492fc9789c0f84794f5fe0d162ca9d042c273bb1bec719f3a0b3b76c8b0ba27796a04c1e4d2cb2d946ac520c2b41589517cb9ae22e64718086c1b13bec1c3d1d78c274d4ffafd78e1b66705e496c0f85a94204fb72a977ff7bac372c05a6948ed717
a46acc8b0677f3dba8acae966b964da81ba1e4d041caf6ae1c45f17e25f0a590698efcc78afd3b5f3d8ab376cfdf0568e55cfbf0cd39206aa3563a6a214a5efb56212e69675920000f85a94204fb72a977ff7bac372c05a6948ed717a46acc8b08e10d248a242edb91b5b40bd4de6fe6da9102c3f86058fbea95288720defcf0ad7a94dc5705f6661
b81cafb83decce0cd39206aa3563a6a214a5efb56212e69675920000f9024c01f90248f8479475ea17db98f7266c89423a4ea12677822ab269bcb04bf54264c1bfa68ca201f756e882f49e1e8aaa5ddf42deaf4690bc3977497e245af40f3ad4003d7a6121614f13033b0bc0f8479436b79e51601704f69219970ef135585257cdcf6fb0f441b7547
0919983ba18a0525b1c101af42cae052c6d50f74d1553eebbe78ef226849c5e5a7fb2ba563eec6b20380c00c0f84794d8c23ff826e00220bfa7ab6966a85998574f1c63b02e9aa982036860eccb0880702c5d71665761f8d4e6ab5f3d8c3aee25b3e68a2c7eaa3cd85972c7f9a3c19d3fed3d5d01c0f8479406992099a65399d243b791b6f0596f8f
00731248b06ed606f323da02b0e18d64a87a1b42641d847c15e9e11596e77d87d34f26679785fb4e76d270dc9d14bb539f02157501c0f847945786636bef546a4a5b276f529432434f6223609db0021926136495a0adcdda5af0602cb4b4ce6d6529cfd451b844728a0e4e424f2a78879a8b5d5e4b3e42127f95f2e2858bc0f8479453fe65ddf3fa2
d19df909df1dbacf109e35d6e9cb03a6c844cd65fa07dbbd6b256a6c04be4f9031aa0043b37e5897a5bbfa35ad32dac3e308e04b17962dff07685e2c2180cc0f8479474ec8294957a0055dc25ba999fef2934278daaa3b04cf88358365733859717ccabc63c73a6ec01a03f07fef266342bfec2f1d57c14b543b9995c3a66b8a737db3a9b60920ac0
f8479428595f3d20cf25b462c0dd2fe04496ba604e8e4ab03a3d6481bb95279254ad74649e65e31a18818c56da38c3623af86e5527cc7bce62f84187f10c9b6c30fd485196501518c08080)
##############################
"}

Environment (please complete the following information):

• OS: [Linux]
• Go environment:

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/uladzislau/.cache/go-build"
GOENV="/home/uladzislau/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/uladzislau/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/uladzislau/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.11"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4176410830=/tmp/go-build -gno-record-gcc-switches"

Additional context
You can check this problem on internal node, just find one stopped on the 20627530 block

[sophoah]

I will add that into my to do list but with a low priority since this is testnet and we can use the snapshot to sync the node.

[darinvhs]

@sophoah, I'm facing a similar issue. Which snapshot and Harmony version can we use for the testnet? I want to resync my node

[mur-me]

Hey @darinvhs, please use:

YOUR_DATA_DIR='./' # This is the default installation, change it you use something else
rclone -P -L --webdav-url 'http://fulldb.s0.b.hmny.io/webdav' \
   --checksum sync snap: "${YOUR_DATA_DIR}harmony_db_0" --multi-thread-streams 4 --transfers=32 --verbose 

Credentials are placed here:
https://docs.harmony.one/home/network/validators/node-setup/syncing-db#id-2.-configuring-rclone

===

This info will be a part of docu above today.

===

Please let me know, if you have questions.

[mur-me]

Hey @darinvhs, and now we have placed it into the docu: https://docs.harmony.one/home/network/validators/node-setup/syncing-db#id-5.1-testnet-shard-0