Cosmwasm implementation produce forks

[RaulBernal]

Describe the bug
After include the WASM module using this official guide and stop/start nodes again, the nodes get Apphashes and fork the chain.

We have tested to:

• include the CosmWasm module during our v0.47>0.50 upgrade
• and also in a separated upgrade from v0.50>v0.50 adding only the wasm module.

In both cases, when one of the two validators stops & starts it get the APPHash.

To reproduce
Steps to reproduce the behavior:

1. Create a Ignite v0.27.x version scaffolded chain and run it with 2 validators
2. Create a Ignite v0.28.3 version scaffolded chain with an upgrade and run it with 2 validators
3. Make the upgrade. It should works.
4. Stop one of two nodes and start again and then it gets forked

This is the log when one of the nodes stops&starts after the successful upgrade of wasm module (doesn't matters if the upgrade happens in the v0.47>v0.50 process or in a separate v0.50>v0.50 module adds

Started BitCanna Node.
11:57AM INF starting node with ABCI CometBFT in-process module=server
11:57AM INF service start impl=multiAppConn module=proxy msg="Starting multiAppConn service"
11:57AM INF service start connection=query impl=localClient module=abci-client msg="Starting localClient service"
11:57AM INF service start connection=snapshot impl=localClient module=abci-client msg="Starting localClient service"
11:57AM INF service start connection=mempool impl=localClient module=abci-client msg="Starting localClient service"
11:57AM INF service start connection=consensus impl=localClient module=abci-client msg="Starting localClient service"
11:57AM INF service start impl=EventBus module=events msg="Starting EventBus service"
11:57AM INF service start impl=PubSub module=pubsub msg="Starting PubSub service"
11:57AM INF service start impl=IndexerService module=txindex msg="Starting IndexerService service"
11:57AM INF ABCI Handshake App Info hash=CFC90057FCE162DB4068505570AD0730909BD6FA8E80E08F7E31841BC5AE72F2 height=831 module=consensus protocol-version=0 software-version=upgrade_image_1-3-g1405633
11:57AM INF ABCI Replay Blocks appHeight=831 module=consensus stateHeight=831 storeHeight=831
11:57AM INF Completed ABCI Handshake - CometBFT and App are synced appHash=CFC90057FCE162DB4068505570AD0730909BD6FA8E80E08F7E31841BC5AE72F2 appHeight=831 module=consensus
11:57AM INF Version info abci=2.0.0 block=11 commit_hash= module=server p2p=8 tendermint_version=0.38.6
11:57AM INF This node is a validator addr=0D61B4BFACE772DA88066EE065B807F0268DDC7D module=consensus pubKey=PubKeyEd25519{2A8D551EEAE3B019ED5031205363D4B8E004B34D62F09795E2DEED78FFF0D170}
11:57AM INF P2P Node ID ID=85be53f499c803a10e73734018334049a7c4686d file=/home/ljn/.bcna/config/node_key.json module=p2p
11:57AM INF Adding persistent peers addrs=["bee4c8fcb34d9e2521f4e30803bd83f3ce90f80d@164.68.119.233:26656"] module=p2p
11:57AM INF Adding unconditional peer ids ids=[] module=p2p
11:57AM INF Add our address to book addr=85be53f499c803a10e73734018334049a7c4686d@0.0.0.0:26656 book=/home/ljn/.bcna/config/addrbook.json module=p2p
11:57AM INF service start impl=Node module=server msg="Starting Node service"
11:57AM INF service start impl="P2P Switch" module=p2p msg="Starting P2P Switch service"
11:57AM INF service start impl=Reactor module=blocksync msg="Starting Reactor service"
11:57AM INF service start impl=BlockPool module=blocksync msg="Starting BlockPool service"
11:57AM INF serve module=rpc-server msg="Starting RPC HTTP server on [::]:26657"
11:57AM INF service start impl=ConsensusReactor module=consensus msg="Starting Consensus service"
11:57AM INF Reactor  module=consensus waitSync=true
11:57AM INF service start impl=Evidence module=evidence msg="Starting Evidence service"
11:57AM INF service start impl=StateSync module=statesync msg="Starting StateSync service"
11:57AM INF service start impl=PEX module=pex msg="Starting PEX service"
11:57AM INF service start book=/home/ljn/.bcna/config/addrbook.json impl=AddrBook module=p2p msg="Starting AddrBook service"
11:57AM INF service start impl=Mempool module=mempool msg="Starting Mempool service"
11:57AM INF Saving AddrBook to file book=/home/ljn/.bcna/config/addrbook.json module=p2p size=2
11:57AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=0 numToDial=10
11:57AM INF starting API server... address=tcp://0.0.0.0:1317 module=api-server
11:57AM INF serve module=api-server msg="Starting RPC HTTP server on [::]:1317"
11:57AM INF starting gRPC server... address=0.0.0.0:9090 module=grpc-server
11:57AM INF Reconnecting to peer addr=bee4c8fcb34d9e2521f4e30803bd83f3ce90f80d@164.68.119.233:26656 module=p2p
11:57AM INF service start impl="Peer{MConn{164.68.119.233:26656} bee4c8fcb34d9e2521f4e30803bd83f3ce90f80d out}" module=p2p msg="Starting Peer service" peer=bee4c8fcb34d9e2521f4e30803bd83f3ce90f80d@164.68.119.233:26656
11:57AM INF service start impl=MConn{164.68.119.233:26656} module=p2p msg="Starting MConnection service" peer=bee4c8fcb34d9e2521f4e30803bd83f3ce90f80d@164.68.119.233:26656
11:57AM INF Time to switch to consensus reactor! height=832 module=blocksync
11:57AM INF service stop impl=BlockPool module=blocksync msg="Stopping BlockPool service"
11:57AM INF SwitchToConsensus module=consensus
11:57AM INF service start impl=ConsensusState module=consensus msg="Starting State service"
11:57AM INF service start impl=baseWAL module=consensus msg="Starting baseWAL service" wal=/home/ljn/.bcna/data/cs.wal/wal
11:57AM INF service start impl=Group module=consensus msg="Starting Group service" wal=/home/ljn/.bcna/data/cs.wal/wal
11:57AM INF service start impl=TimeoutTicker module=consensus msg="Starting TimeoutTicker service"
11:57AM INF Searching for height height=832 max=0 min=0 module=consensus wal=/home/ljn/.bcna/data/cs.wal/wal
11:57AM INF Searching for height height=831 max=0 min=0 module=consensus wal=/home/ljn/.bcna/data/cs.wal/wal
11:57AM INF Found height=831 index=0 module=consensus wal=/home/ljn/.bcna/data/cs.wal/wal
11:57AM INF Catchup by replaying consensus messages height=832 module=consensus
11:57AM INF Replay: New Step height=832 module=consensus round=0 step=RoundStepNewHeight
11:57AM INF Replay: Done module=consensus
11:57AM INF received proposal module=consensus proposal="Proposal{832/0 (CABE80CC2AA8D80D8DA52534A7B8024414FFEF8A02800EFA632D4308EA3581F7:1:C252DE7E772E, -1) EABC383478C7 @ 2024-05-10T11:55:15.821754105Z}" proposer=3332A1D571F4F0489C4168A9075DA71F1A764945
11:57AM INF received complete proposal block hash=CABE80CC2AA8D80D8DA52534A7B8024414FFEF8A02800EFA632D4308EA3581F7 height=832 module=consensus
11:57AM INF finalizing commit of block hash=CABE80CC2AA8D80D8DA52534A7B8024414FFEF8A02800EFA632D4308EA3581F7 height=832 module=consensus num_txs=0 root=CFC90057FCE162DB4068505570AD0730909BD6FA8E80E08F7E31841BC5AE72F2
11:57AM INF finalized block block_app_hash=99945E19DE31222B4145E4102B5128D63342E4E2E2B4E4BC862F8239C2F75694 height=832 module=state num_txs_res=0 num_val_updates=0
11:57AM INF executed block app_hash=99945E19DE31222B4145E4102B5128D63342E4E2E2B4E4BC862F8239C2F75694 height=832 module=state
11:57AM INF committed state block_app_hash=CFC90057FCE162DB4068505570AD0730909BD6FA8E80E08F7E31841BC5AE72F2 height=832 module=state
11:57AM INF indexed block events height=832 module=txindex
11:57AM INF Timed out dur=4988.041822 height=833 module=consensus round=0 step=RoundStepNewHeight
11:57AM INF received proposal module=consensus proposal="Proposal{833/0 (57286F3C5623E1DBEA877E52FE1127C3B55A3FA9614BBDFDC212741C5ECC7363:1:C5C6520CEADB, -1) 5535BAFCBA01 @ 2024-05-10T11:57:15.111821552Z}" proposer=0D61B4BFACE772DA88066EE065B807F0268DDC7D
11:57AM INF received complete proposal block hash=57286F3C5623E1DBEA877E52FE1127C3B55A3FA9614BBDFDC212741C5ECC7363 height=833 module=consensus
11:57AM INF Timed out dur=1000 height=833 module=consensus round=0 step=RoundStepPrevoteWait
11:57AM INF Timed out dur=1000 height=833 module=consensus round=0 step=RoundStepPrecommitWait
11:57AM INF resetting proposal info height=833 module=consensus proposer=3332A1D571F4F0489C4168A9075DA71F1A764945 round=1
11:57AM INF received proposal module=consensus proposal="Proposal{833/1 (B31694459C1B7288E4B86A18BBAD13B69D59B39FA997DBC6C9A4728AF1FBDF10:1:437A80B6C03C, -1) 7D8A84FA7922 @ 2024-05-10T11:57:17.43530084Z}" proposer=3332A1D571F4F0489C4168A9075DA71F1A764945
11:57AM INF received complete proposal block hash=B31694459C1B7288E4B86A18BBAD13B69D59B39FA997DBC6C9A4728AF1FBDF10 height=833 module=consensus
11:57AM ERR prevote step: consensus deems this block invalid; prevoting nil err="wrong Block.Header.AppHash.  Expected 99945E19DE31222B4145E4102B5128D63342E4E2E2B4E4BC862F8239C2F75694, got A1EB97CB19E5C79C042F01C730AEF74FCD22B93C55FE62361378B87EC9C1EB4E" height=833 module=consensus round=1

What version are you using?
v0.28.3

Source code tested

For reference, this is the last code used to make a test with v0.50.5 to v0.50.5 adding the wasm module that produce forks (not in the upgrade time, but after)

• WASM integration (same that Ignite add wasm): RaulBernal/bcna_v4cosmwasm@ec15912
• Upgrade handler: RaulBernal/bcna_v4cosmwasm@2a4c87b

[RaulBernal]

Maybe upgrade handler is incomplete in that case; to check:

• https://github.com/CosmWasm/wasmd/blob/v0.50.0/app/upgrades.go
• https://github.com/CosmWasm/wasmd/blob/v0.51.0/app/upgrades.go

[RaulBernal]

Doesn't help with forks (tested)
https://github.com/RaulBernal/bcna_v4cosmwasm/pull/1/files

[RaulBernal]

Also fixed the wasm path but still forks
RaulBernal/bcna_v4cosmwasm@6abefb8

[RaulBernal]

I've tested the changes proposed here (cosmos/cosmos-sdk#18521) and also it forks

[RaulBernal]

FYI I've tested to make a no-op upgrade and it doesn't forks; later apply a new upgrade with wasm's adding and it forked

[Pantani]

@RaulBernal, check the wasm implementation from the ignite wasm app

[RaulBernal]

@RaulBernal, check the wasm implementation from the ignite wasm app
Hi @Pantani as I described at the beginning of the issue, I follow this official guide that reproduces the same commands for install wasm that ignite wasm app
Can you point me to the differences please?

[Pantani]

@RaulBernal can you share your chain config TOML? If you restart your chain with Ignite using the -r flag maybe it can overwrite your chain TOML config

[RaulBernal]

@RaulBernal can you share your chain config TOML? If you restart your chain with Ignite using the -r flag maybe it can overwrite your chain TOML config

I've checked my config.toml in devchain and it hasn't had the Wasm config lines.
I added them to config.toml:

[wasm]
# Smart query gas limit is the max gas to be used in a smart query contract call
query_gas_limit = 3000000

# in-memory cache for Wasm contracts. Set to 0 to disable.
# The value is in MiB not bytes
memory_cache_size = 100

# Simulation gas limit is the max gas to be used in a tx simulation call.
# When not set the consensus max block gas is used instead
# simulation_gas_limit =

and this to app.toml:

[wasm]
# Smart query gas limit is the max gas to be used in a smart query contract call
query_gas_limit = 3000000

I restored a backup and performed new upgrades, from v0.47 to v0.50 and add the Wasm module at v0.50 and it forks again :(

10:25PM ERR CONSENSUS FAILURE!!! err="+2/3 committed an invalid block: wrong Block.Header.AppHash. Expected EE9B6FA3A3C895993A183BE37EEB42E6D67A57EB285E14ED5163E7DC85956D08, got E5056C2DD23A42B6D1815383323EC8C8ADF963366FA50E88DCB9384D14EF6DEE" module=consensus stack="goroutine 171 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\t/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:801 +0x46\npanic({0x2bcdc80?, 0xc002a4e5c0?})\n\t/usr/local/go/src/runtime/panic.go:914 +0x21f\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc003004000, 0x2a4)\n\t/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:1713 +0x106e\ngithub.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0xc003004000, 0x2a4)\n\t/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:1682 +0x2f6\ngithub.com/cometbft/cometbft/consensus.(*State).handleCompleteProposal(0xc003004000, 0xc002a1a900?)\n\t/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:2060 +0x37d\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg(0xc003004000, {{0x3ee3aa0, 0xc001a1bce0}, {0xc0034da000, 0x28}})\n\t/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:910 +0x1a5\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine(0xc003004000, 0x0)\n\t/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:836 +0x3d1\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutine 132\n\t/home/raul/go/pkg/mod/github.com/cometbft/cometbft@v0.38.6/consensus/state.go:398 +0x10c\n"