Hermes queries invalid (future) block heights

[michaelfig]

refs: #3299

Summary of Bug

The Agoric chain has a small patch to assert queries are only made for blocks that have been produced. It sends a "given height %d is greater than latest height %d: invalid height" error if a query for a future block is made. It seems if this error happens, the transfer channel gets stuck.

Version

Official docker version for hermes 1.8.0.

Steps to Reproduce

1. Launch Hermes with the following config:

config.toml

# The global section has parameters that apply globally to the relayer operation.
[global]

log_level = 'info'


# Specify the mode to be used by the relayer. [Required]
[mode]

# Specify the client mode.
[mode.clients]

# Whether or not to enable the client workers. [Required]
enabled = true

# Whether or not to enable periodic refresh of clients. [Default: true]
# This feature only applies to clients that underlie an open channel.
# For Tendermint clients, the frequency at which Hermes refreshes them is 2/3 of their
# trusting period (e.g., refresh every ~9 days if the trusting period is 14 days).
# Note: Even if this is disabled, clients will be refreshed automatically if
#      there is activity on a connection or channel they are involved with.
refresh = true

# Whether or not to enable misbehaviour detection for clients. [Default: false]
misbehaviour = false

# Specify the connections mode.
[mode.connections]

# Whether or not to enable the connection workers for handshake completion. [Required]
enabled = false

# Specify the channels mode.
[mode.channels]

# Whether or not to enable the channel workers for handshake completion. [Required]
enabled = false

# Specify the packets mode.
[mode.packets]

# Whether or not to enable the packet workers. [Required]
enabled = true

# Parametrize the periodic packet clearing feature.
# Interval (in number of blocks) at which pending packets
# should be periodically cleared. A value of '0' will disable
# periodic packet clearing. [Default: 100]
clear_interval = 100

# Whether or not to clear packets on start. [Default: false]
clear_on_start = true

# Toggle the transaction confirmation mechanism.
# The tx confirmation mechanism periodically queries the `/tx_search` RPC
# endpoint to check that previously-submitted transactions
# (to any chain in this config file) have been successfully delivered.
# If they have not been, and `clear_interval = 0`, then those packets are
# queued up for re-submission.
# If set to `false`, the following telemetry metrics will be disabled:
# `acknowledgment_packets_confirmed`, `receive_packets_confirmed` and `timeout_packets_confirmed`.
# [Default: false]
tx_confirmation = false

# The REST section defines parameters for Hermes' built-in RESTful API.
# https://hermes.informal.systems/rest.html
[rest]

# Whether or not to enable the REST service. Default: false
enabled = true

# Specify the IPv4/6 host over which the built-in HTTP server will serve the RESTful
# API requests. Default: 127.0.0.1
host = '127.0.0.1'

# Specify the port over which the built-in HTTP server will serve the restful API
# requests. Default: 3000
port = 3000


# The telemetry section defines parameters for Hermes' built-in telemetry capabilities.
# https://hermes.informal.systems/telemetry.html
[telemetry]

# Whether or not to enable the telemetry service. Default: false
enabled = false

# Specify the IPv4/6 host over which the built-in HTTP server will serve the metrics
# gathered by the telemetry service. Default: 127.0.0.1
host = '127.0.0.1'

# Specify the port over which the built-in HTTP server will serve the metrics gathered
# by the telemetry service. Default: 3001
port = 3001


# A chains section includes parameters related to a chain and the full node to which
# the relayer can send transactions and queries.

[[chains]]
id = 'agoric-emerynet-8'
rpc_addr = 'https://emerynet.rpc.agoric.net/'
grpc_addr = 'https://emerynet.grpc.agoric.net/'
event_source = { mode = 'push', url = 'wss://emerynet.rpc.agoric.net/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'agoric'
key_name = 'whale-emerynet'
address_type = { derivation = 'cosmos' }
store_prefix = 'ibc'
default_gas = 2000000
max_gas = 10000000
gas_price = { price = 0.00, denom = 'ubld' }
gas_multiplier = 1.1
max_msg_num = 25
max_tx_size = 180000
clock_drift = '15s'
#max_block_time = '15s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }
memo_prefix = 'agoric-emerynet-relay'

[[chains]]
id = 'agoricdev-23'
rpc_addr = 'https://devnet.rpc.agoric.net/'
grpc_addr = 'https://devnet.grpc.agoric.net/'
event_source = { mode = 'push', url = 'wss://devnet.rpc.agoric.net:26657/websocket', batch_delay = '500ms' }
rpc_timeout = '15s'
account_prefix = 'agoric'
key_name = 'whale-devnet'
address_type = { derivation = 'cosmos' }
store_prefix = 'ibc'
default_gas = 2000000
max_gas = 10000000
gas_price = { price = 0.00, denom = 'ubld' }
gas_multiplier = 1.1
max_msg_num = 25
max_tx_size = 180000
clock_drift = '15s'
#max_block_time = '15s'
trusting_period = '14days'
trust_threshold = { numerator = '1', denominator = '3' }
memo_prefix = 'agoric-devnet-relay'

2. Get ubld from https://emerynet.faucet.agoric.net and https://devnet.faucet.agoric.net (send BLD/IBC toy tokens)
3. Transfer ubld from emerynet to devnet using the relayer addresses. This will usually work (but sometimes fails as below logs indicate).
4. Transfer the ibc/XXX vouchers from devnet to emerynet. This will almost always fail as below logs indicate.

Here is the invalid height WARN I see sent by Hermes (all in one line):

2024-03-04T18:18:01.833958Z  WARN ThreadId(99) worker.batch{chain=agoric-emerynet-8}
:supervisor.handle_batch{chain=agoric-emerynet-8}:supervisor.process_batch{chain=agoric-emerynet-8}
:worker.packet.cmd{src_chain=agoric-emerynet-8 src_port=transfer src_channel=channel-23
 dst_chain=agoricdev-23}: task encountered ignorable error: link error: channel error
: failed during a query to chain 'agoricdev-23': ABCI query returned an error: AbciQuery
{ code: Err(26), log: "given height 2217615 is greater than latest height 2217614: invalid height",
 info: "", index: 0, key: [], value: [], proof: None, height: block::Height(0), codespace: "sdk" }

and later:

2024-03-04T18:26:36.232334Z ERROR ThreadId(18) spawn:chain{chain=agoricdev-23}
:client{client=07-tendermint-19}:connection{connection=connection-18}
:channel{channel=channel-16}:worker.client.refresh{client=07-tendermint-19
 src_chain=agoric-emerynet-8 dst_chain=agoricdev-23}:foreign_client
.refresh{client=agoric-emerynet-8->agoricdev-23:07-tendermint-19}:foreign_client
.build_update_client_and_send{client=agoric-emerynet-8->agoricdev-23:07-tendermint-19
 target_query_height=latest height}
:send_messages_and_wait_commit{chain=agoricdev-23 tracking_id=update client}
:send_tx_with_account_sequence_retry{chain=agoricdev-23 account.sequence=215}
:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call `send_tx_simulate`
 failed with status: status: Unknown, message: "account sequence mismatch, expected 216, got 215
: incorrect account sequence
 [agoric-labs/cosmos-sdk@v0.45.16-alpha.agoric.3/x/auth/ante/sigverify.go:265]
 With gas wanted: '0' and gas used: '37213' "

Acceptance Criteria

Ensure that future block heights are not queried. Clear guidance as to whether this is part of the "account sequence mismatch" issue.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[romac]

How often do you see this? Could you run hermes start with --debug=rpc for while and see if you run into this again?

$ hermes --debug=rpc start