"Account sequence mismatch" with the same sequence multiple times

[freak12techno]

Summary of Bug

Getting a lot of "Account sequence mismatch" errors on osmosis-1 <=> cosmoshub-4 channel with the same sequence:

• something is submitted
• the logs are showing the error with account sequence mismatch
• something else is happening (not sure what can influence it, but in my case it was a tx that was sent but its confirmation has timed out apparently)
• something else is submitted
• the logs are showing the same error with the same sequence (although it should've been updated once it failed?)

I guess when timing out it assumes that the tx went through and increases sequence while in fact the tx failed, but I may be wrong here.

Version

v1.8.0, although I've faced it a lot on 1.7.4 as well.

Steps to Reproduce

• Run Hermes on cosmoshub-4 <=> osmosis-1 channel
• not sure what else can be relevant here

Logs illustrating the issue (I omitted some not relevant to this issue, let me know if you need all of them):

2024-01-25T19:49:34.232408Z  WARN ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}:foreign_client.wait_and_build_update_client_with_trusted{client=cosmoshub-4->osmosis-1:07-tendermint-1 target_height=4-18877181}:foreign_client.build_update_client_with_trusted{client=cosmoshub-4->osmosis-1:07-tendermint-1 target_height=4-18877181}:foreign_client.solve_trusted_height{client=cosmoshub-4->osmosis-1:07-tendermint-1 target_height=4-18877181}: resolving trusted height from the full list of consensus state heights for target height 4-18877181; this may take a while
2024-01-25T19:49:40.968061Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}: assembled batch of 4 message(s)
2024-01-25T19:49:41.553307Z  WARN ThreadId(25) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=ca44850d}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=164}:estimate_gas: failed to simulate tx, falling back on default gas because the error is potentially recoverable: gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 163, got 164: incorrect account sequence [cosmos/cosmos-sdk@v0.47.5/x/auth/ante/sigverify.go:269] With gas wanted: '300000000' and gas used: '1040081' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "13449501"} }
2024-01-25T19:49:41.802723Z  WARN ThreadId(25) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=ca44850d}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=164}: failed to broadcast tx because of a mismatched account sequence number, refreshing account sequence number and retrying once response=Response { code: Err(32), data: b"", log: "account sequence mismatch, expected 163, got 164: incorrect account sequence", hash: Hash::Sha256(D73369273F639559318AE388F6877F8CE03BA027B042CFC974DB3F3B7983E554) }
2024-01-25T19:49:42.036170Z  INFO ThreadId(25) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=ca44850d}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=164}: refreshed account sequence number old=164 new=163
2024-01-25T19:49:43.432560Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}: response(s): 1; Error with code 22:D3BE52D57BD5C52AA5DB741E4B5D33F22A47548CAF64FC822BE9FB68D0CACF85 target_chain=osmosis-1



2024-01-25T19:49:43.432595Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=ca44850d ->Destination @4-18877180; len=3}: submitted
2024-01-25T19:49:56.327062Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=e1aed966 ->Destination @4-18877185; len=1}: assembled batch of 2 message(s)
2024-01-25T19:49:57.751350Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=e1aed966 ->Destination @4-18877185; len=1}: response(s): 1; Ok:822A1D5CE4780665562EC62F4D27B3304582ED0034057A2F331FF5FE09226FF9 target_chain=osmosis-1


2024-01-25T19:49:57.751383Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=e1aed966 ->Destination @4-18877185; len=1}: submitted
2024-01-25T19:50:03.251855Z  INFO ThreadId(31) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=c764403c ->Destination @4-18877186; len=2}: assembled batch of 3 message(s)
2024-01-25T19:50:04.664632Z  WARN ThreadId(25) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=c764403c ->Destination @4-18877186; len=2}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=c764403c}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=164}: failed to broadcast tx because of a mismatched account sequence number, refreshing account sequence number and retrying once response=Response { code: Err(32), data: b"", log: "account sequence mismatch, expected 163, got 164: incorrect account sequence", hash: Hash::Sha256(8CCADFE27612EF2FF6859D35FC241146630B4E5C8DE2B5A649885EB286E68324) }
2024-01-25T19:50:04.902131Z  INFO ThreadId(25) spawn:chain{chain=cosmoshub-4}:client{client=07-tendermint-259}:connection{connection=connection-257}:channel{channel=channel-141}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=c764403c ->Destination @4-18877186; len=2}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=c764403c}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=164}: refreshed account sequence number old=164 new=163

Acceptance Criteria

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[freak12techno]

Might be the same issue: #3547

[adizere]

Hey @freak12techno is this still happening regularly?

[freak12techno]

@adizere I guess

Apr 02 23:46:59 monitoring-logging hermes[634298]: 2024-04-02T21:46:59.002990Z  WARN ThreadId(25) worker.batch{chain=cosmoshub-4}:supervisor.handle_batch{chain=cosmoshub-4}:supervisor.process_batch{chain=cosmoshub-4}:worker.packet.cmd{src_chain=cosmoshub-4 src_port=transfer src_channel=channel-141 dst_chain=osmosis-1}:relay{odata=35f843ed ->Destination @4-19830002; len=1}:send_messages_and_wait_check_tx{chain=osmosis-1 tracking_id=35f843ed}:send_tx_with_account_sequence_retry{chain=osmosis-1 account.sequence=6071}:estimate_gas: failed to simulate tx, falling back on default gas because the error is potentially recoverable: gRPC call `send_tx_simulate` failed with status: status: Unknown, message: "account sequence mismatch, expected 6070, got 6071: incorrect account sequence [cosmos/cosmos-sdk@v0.47.8/x/auth/ante/sigverify.go:269] With gas wanted: '300000000' and gas used: '831344' ", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "14653501"} }