Hermes ignores some ICS-20 packets for some reason (on osmosis, CW contract creates packet as called back by incoming Wasm hook)

[dzmitry-lahoda]

Summary of Bug

I run Hermes amid Centauri and Osmosis.
It successfully relays from Centauri to Osmosis all the time.

But from Osmosis to Centauri it has some troubles.

Not not always the case, but more often packets are not relayed when ICS-20 packet has memo and was created as part of CW transaction on Osmosis.

With ICS-20 transfer via Csmosis cli, sometimes packets stuck, but not so often.

I checked that logs has ibc transfer and escrow locks funds, so I am sure(as sure I can be) on chain code is correct.

I set logs to trace, do not see any errors. Also have set bigger TX limits too.

Version

I use cosmos.nix to run all nodes.

Steps to Reproduce

Setup nix with flakes and nix-command enabled. https://docs.composable.finance/nix/install/

git clone https://github.com/ComposableFi/composable.git && cd composable

Run

nix run .#devnet-xc-cosmos-fresh

and wait when hermes relayer running along with centauri and osmosis chains

Than run nix run .#xc-swap-pica-to-osmo - it will send in loop CW execute:

1. Invoke CW on Centauri
2. Form ICS-20 transfer with memo.
3. Memo will arrive Osmosis
4. Call CW
5. CW will send ICS-20 packet to Centauri
   Expected: Packet reaches Centauri
   Actual: Packet never reaches Centauri

I attached logs of nodes nodes and relayer.

What I have checked?

1. Logs show that packet was sent from Centauri and reached Osmosis.
2. Logs show that IBC fungible packet was sent from Osmosis, packet from 1 acked.
3. Funds sent from Osmosis escrowed in IBC (can check via osmosisd commands of transfer channel-0) escrow.
4. But packet is never delivered to Centauri and I do not see logs in Hermses about attemps to deliver.

All logs are set to debug and/or trace.

logs.zip

Also I send some funds from osmosis osmosisd tx ibc-transfer transfer "transfer" "channel-0" "centauri1qvdeu4x34rapp3wc8fym5g4wu343mswxxgc6wf" 12345678uosmo --from=1 --fees=100000uosmo -y and packets sometimes delivered. But when I send packet with memo, it is like stuck.

Versions

(devenv) bash-5.2$ nix run .#osmosisd version
v18.0.0
(devenv) bash-5.2$ nix run .#centaurid version
v4.5.0
(devenv) bash-5.2$ nix run .#hermes version
2023-09-20T11:29:49.029662Z  INFO ThreadId(01) running Hermes v1.6.0
hermes 1.6.0
(devenv) bash-5.2$ 
(devenv) bash-5.2$ uname -a
Linux pop-os 5.19.0-76051900-generic #202207312230~1663791054~22.04~28340d4 SMP PREEMPT_DYNAMIC Wed S x86_64 x86_64 x86_64 GNU/Linux

Acceptance Criteria

More debugging on reasons of possible packet ignorance or transaction failures in hermes.

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[dzmitry-lahoda]

here is video of same contracts working well on mainnet https://youtu.be/kxLkKzYW2xw with hermes. UP: fixing low res video

so sure things work, but hermese for some reason reports no warnings or errors locally.

[romac]

Thanks for the detailed report! 🙏

Which event source are you using, push or pull? If you are not already using the pull mode, can you try with event_source = { mode = 'pull', url = '...' } and report back?

Could you also please post your config.toml here?

[dzmitry-lahoda]

i use this config

[global]
log_level = 'trace'
[mode]
[mode.clients]
enabled = true
refresh = true
misbehaviour = false
[mode.connections]
enabled = true
[mode.channels]
enabled = true

[mode.packets]
enabled = true
clear_interval = 0
clear_on_start = false
tx_confirmation = true

[rest]
enabled = false
host = '127.0.0.1'
port = 30042

[telemetry]
enabled = false
host = '127.0.0.1'
port = 30041

[[chains]]
id = 'centauri-dev'
rpc_addr = 'http://127.0.0.1:26657'
grpc_addr = 'http://127.0.0.1:9090'
#event_source = { mode = 'push', url = 'ws://127.0.0.1:26657/websocket', batch_delay = '1000ms' }
event_source = { mode = 'pull', interval = '1s' }
rpc_timeout = '30s'
account_prefix = 'centauri'
key_name = 'centauri-dev'
store_prefix = 'ibc'
default_gas = 100000000
max_gas = 40000000000
gas_price = { price = 1, denom = 'ppica' }
gas_multiplier = 1.3
max_msg_num = 5
max_tx_size = 4097152
clock_drift = '10s'
max_block_time = '30s'
trusting_period = '640s'
trust_threshold = { numerator = '1', denominator = '3' }
type = 'CosmosSdk'
address_type = { derivation = 'cosmos' }
trusted_node = true
key_store_type = 'Test'

[[chains]]
id = 'osmosis-dev'
rpc_addr = 'http://127.0.0.1:36657'
grpc_addr = 'http://127.0.0.1:19090'
#event_source = { mode = 'push', url = 'ws://127.0.0.1:36657/websocket', batch_delay = '1000ms' }
event_source = { mode = 'pull', interval = '1s' }
rpc_timeout = '20s'
account_prefix = 'osmo'
key_name = 'osmosis-dev'
store_prefix = 'ibc'
key_store_type = 'Test'
default_gas = 10000000
max_gas = 4000000000
gas_price = { price = 1, denom = 'uosmo' }
gas_multiplier = 1.1
max_msg_num = 5
max_tx_size = 4097152
clock_drift = '10s'
max_block_time = '30s'
trusting_period = '640s'
trust_threshold = { numerator = '1', denominator = '3' }
type = 'CosmosSdk'
address_type = { derivation = 'cosmos' }
trusted_node = true

[dzmitry-lahoda]

As I see I already use pull, so it works without explicit url config. I think issue may be some config of trusting period/validators rotations timeouts, so not errors in logs about this. Alternatively I think to restart hermes several times - if it will catch up after restart, did not tried.