ERROR failed to spawn chain runtime: relayer error: RPC error to endpoint

[taitruong]

Running Hermes 1.6.0. This error appears only when doing 2 hermes calls:

1. hermes start
2. hermes clear packets

Calling 1) or 2) works fine. You can test it like this: run 1) hermes start and then call 2) a couple of times hermes --config ./config.toml clear packets --chain uni-6 --channel channel-721 --port wasm.juno1uvflayc34fs9r3tzcausztks2058ecpt5kqtzf3sz5kwnu7fjtfqn7zmtj

Then you get this error:

ERROR failed to spawn chain runtime: relayer error: RPC error to endpoint https://juno-testnet-rpc.polkachu.com/: serde parse error: expected value at line 1 column 1

For testing use this config.toml:

# Description of the parameters below can be found here: https://hermes.informal.systems/documentation/configuration/description.html
# The global section has parameters that apply globally to the relayer operation.
[global]

# Specify the verbosity for the relayer logging output. Default: 'info'
# Valid options are 'error', 'warn', 'info', 'debug', 'trace'.
log_level = 'info'

# Specify the mode to be used by the relayer. [Required]
[mode]

# Specify the client mode.
[mode.clients]

# Whether or not to enable the client workers. [Required]
enabled = true

# Whether or not to enable periodic refresh of clients. [Default: true]
# This feature only applies to clients that underlie an open channel.
# For Tendermint clients, the frequency at which Hermes refreshes them is 2/3 of their
# trusting period (e.g., refresh every ~9 days if the trusting period is 14 days).
# Note: Even if this is disabled, clients will be refreshed automatically if
#      there is activity on a connection or channel they are involved with.
refresh = true

# Whether or not to enable misbehaviour detection for clients. [Default: true]
misbehaviour = true

# Specify the connections mode.
[mode.connections]

# Whether or not to enable the connection workers for handshake completion. [Required]
enabled = true

# Specify the channels mode.
[mode.channels]

# Whether or not to enable the channel workers for handshake completion. [Required]
enabled = true

# Specify the packets mode.
[mode.packets]

# Whether or not to enable the packet workers. [Required]
enabled = true

# Parametrize the periodic packet clearing feature.
# Interval (in number of blocks) at which pending packets
# should be periodically cleared. A value of '0' will disable
# periodic packet clearing. [Default: 100]
clear_interval = 0 # set to 0, for requeueing packets as defined below in `tx_confirmation`

# Whether or not to clear packets on start. [Default: true]
clear_on_start = true

# Toggle the transaction confirmation mechanism.
# The tx confirmation mechanism periodically queries the `/tx_search` RPC
# endpoint to check that previously-submitted transactions
# (to any chain in this config file) have been successfully delivered.
# If they have not been, and `clear_interval = 0`, then those packets are
# queued up for re-submission.
# If set to `false`, the following telemetry metrics will be disabled:
# `acknowledgment_packets_confirmed`, `receive_packets_confirmed` and `timeout_packets_confirmed`.
# [Default: false]
tx_confirmation = true

# Auto register the counterparty payee on a destination chain to
# the relayer's address on the source chain. This can be used
# for simple configuration of the relayer to receive fees for
# relaying RecvPacket on fee-enabled channels.
# For more complex configuration, turn this off and use the CLI
# to manually register the payee addresses.
# [Default: false]
auto_register_counterparty_payee = false

# The REST section defines parameters for Hermes' built-in RESTful API.
# https://hermes.informal.systems/rest.html
[rest]

# Whether or not to enable the REST service. Default: false
enabled = true

# Specify the IPv4/6 host over which the built-in HTTP server will serve the RESTful
# API requests. Default: 127.0.0.1
host = '0.0.0.0' # need to set this to `0.0.0.0` to allow external access within docker! check here: https://github.com/informalsystems/hermes/issues/3592#issuecomment-1712645510

# Specify the port over which the built-in HTTP server will serve the restful API
# requests. Default: 3000
port = 3000


# The telemetry section defines parameters for Hermes' built-in telemetry capabilities.
# https://hermes.informal.systems/telemetry.html
[telemetry]

# Whether or not to enable the telemetry service. Default: false
enabled = true

# Specify the IPv4/6 host over which the built-in HTTP server will serve the metrics
# gathered by the telemetry service. Default: 127.0.0.1
host = '0.0.0.0' # need to set this to `0.0.0.0` to allow external access within docker! check here: https://github.com/informalsystems/hermes/issues/3592#issuecomment-1712645510

# Specify the port over which the built-in HTTP server will serve the metrics gathered
# by the telemetry service. Default: 3001
port = 3001

[telemetry.buckets.latency_submitted]
start = 500
end = 20000
buckets = 10

[telemetry.buckets.latency_confirmed]
start = 1000
end = 30000
buckets = 10

[[chains]]
id = 'elgafar-1'
type = 'CosmosSdk'
rpc_addr = 'https://rpc.elgafar-1.stargaze-apis.com:443' # 'https://stargaze-rpc.polkachu.com:443' 'https://rpc.elgafar-1.stargaze-apis.com:443'
grpc_addr = 'http://grpc-1.elgafar-1.stargaze-apis.com:26660' # 'http://stargaze-grpc.polkachu.com:13790' # 'http://grpc-1.elgafar-1.stargaze-apis.com:26660'
rpc_timeout = '10s'
account_prefix = 'stars'
key_name = 'stargaze_ark_relayer'
store_prefix = 'ibc'
default_gas = 40000
max_gas = 1500000
gas_multiplier = 1.3
max_msg_num = 10
max_tx_size = 4194304 # 4MiB
clock_drift = '60s'
max_block_time = '30s'
memo_prefix = 'Powered by Ark Protocol'
# trusting_period = '1209599s' # 14 days (unbonding period) minus 1 sec
# address_type = { derivation = 'cosmos' }

[chains.event_source]
mode = 'push'
url = 'wss://rpc.elgafar-1.stargaze-apis.com:443/websocket' # 'wss://stargaze-rpc.polkachu.com:443/websocket' 'wss://rpc.elgafar-1.stargaze-apis.com:443/websocket'
batch_delay = '500ms'

[chains.trust_threshold]
numerator = '1'
denominator = '3'

[chains.gas_price]
price = 0.04
denom = 'ustars'

[chains.packet_filter]
policy = 'allow'
list = [
  # Stargaze ICS721 channels
  # IMPORTANT: do NOT change comments for each channel below! `create-channels.sh` does a search and replace!
  ## ICS721 with rate limiter outgoing proxy (managed by Ark)
  ['wasm.stars165fr5j3ea0dk5yr5sr4h8uw76z5fd34lpnqc2x80ya9xkx9u0k3qjrqpzq', 'channel-589'], # CHANNEL_1_STARGAZE_TO_JUNO, channel-721
  ['wasm.stars165fr5j3ea0dk5yr5sr4h8uw76z5fd34lpnqc2x80ya9xkx9u0k3qjrqpzq', 'channel-590'], # CHANNEL_1_STARGAZE_TO_TERRA, channel-354
]

[[chains]]
id = 'uni-6'
rpc_addr = 'https://uni-rpc.reece.sh:443' # 'https://juno-testnet-rpc.polkachu.com:443' # 'https://uni-rpc.reece.sh:443' #'https://rpc.uni.junonetwork.io:443' 'http://juno.rpc.t.stavr.tech:1067'
grpc_addr = 'http://juno-testnet-grpc.polkachu.com:12690' # 'http://juno.grpc.t.stavr.tech:504'
# BACKUP:
# rpc_addr = 'https://uni-rpc.reece.sh:443'
# grpc_addr = 'http://5.161.80.115:9090'
rpc_timeout = '10s'
account_prefix = 'juno'
key_name = 'juno_ark_relayer'
store_prefix = 'ibc'
default_gas = 4500
max_gas = 800000
gas_multiplier = 1.5
max_msg_num = 10
max_tx_size = 4194304 # 4MiB
clock_drift = '60s'
max_block_time = '30s'
memo_prefix = 'Powered by Ark Protocol'
# trusting_period = '2419199s' # 28 days (unbonding period) minus 1 sec
# address_type = { derivation = 'cosmos' }

[chains.event_source]
mode = 'push'
url = 'wss://uni-rpc.reece.sh:443/websocket'
# url = 'wss://rpc.uni.junonetwork.io/websocket'
# url = 'wss://juno-testnet-rpc.polkachu.com:443/websocket' # not supported by :( answer here: https://discord.com/channels/888644988584460369/954902712401661993/1051772830112546896
# url = 'wss://http://juno.rpc.t.stavr.tech:1067/websocket' # not supported
batch_delay = '500ms'

[chains.trust_threshold]
numerator = '1'
denominator = '3'

[chains.gas_price]
price = 0.0045
denom = 'ujunox'

[chains.packet_filter]
policy = 'allow'
list = [
  # Juno ICS721 channels
  # IMPORTANT: do NOT change comments for each channel below! `create-channels.sh` does a search and replace!
  ## ICS721 with rate limiter outgoing proxy (managed by Ark)
  ['wasm.juno1uvflayc34fs9r3tzcausztks2058ecpt5kqtzf3sz5kwnu7fjtfqn7zmtj', 'channel-721'], # CHANNEL_1_JUNO_TO_STARGAZE, channel-589
  ['wasm.juno1uvflayc34fs9r3tzcausztks2058ecpt5kqtzf3sz5kwnu7fjtfqn7zmtj', 'channel-722'], # CHANNEL_1_JUNO_TO_TERRA, channel-353
]

[[chains]]
id = 'pisco-1'
rpc_addr = 'https://terra-testnet-rpc.polkachu.com:443'
grpc_addr = 'http://terra-testnet-grpc.polkachu.com:11790'
# check URLs on deployed terra node here: https://deploy.cloudmos.io/deployments
# rpc_addr = 'https://v6o9a68htpb370gnm6qps5o5ok.ingress.europlots.com'
# grpc_addr = 'http://provider.europlots.com:30583' # check public port mapping for internal port 9090 defined on cloudmos.io
rpc_timeout = '20s'
account_prefix = 'terra'
key_name = 'terra_ark_relayer'
store_prefix = 'ibc'
default_gas = 15000
max_gas = 600000
gas_multiplier = 1.3
max_msg_num = 30
max_tx_size = 100000 # 1k
clock_drift = '45s'
max_block_time = '10s'
memo_prefix = 'Powered by Ark Protocol'
# trusting_period = '288000s'
# address_type = { derivation = 'cosmos' }

[chains.event_source]
mode = 'push'
url = 'wss:///pisco-rpc.terra.dev:443/websocket'
# url = 'ws://terra-testnet-rpc.polkachu.com:443/websocket' # not supported by polkachu, answer here: https://discord.com/channels/888644988584460369/954902712401661993/1051772830112546896
batch_delay = '500ms'

[chains.trust_threshold]
numerator = '1'
denominator = '3'

[chains.gas_price]
price = 0.15
denom = 'uluna'

[chains.packet_filter]
policy = 'allow'
list = [
  # Terra ICS721 channels
  # IMPORTANT: do NOT change comments for each channel below! `create-channels.sh` does a search and replace!
  ## ICS721 with rate limiter outgoing proxy (managed by Ark)
  ['wasm.terra1luhy2ngj7y0lqy3vjek3xm777pywptzzea0sc5ymu4pmafjrn07s5rlmxt', 'channel-353'], # CHANNEL_1_TERRA_TO_JUNO, channel-722
  ['wasm.terra1luhy2ngj7y0lqy3vjek3xm777pywptzzea0sc5ymu4pmafjrn07s5rlmxt', 'channel-354'], # CHANNEL_1_TERRA_TO_STARGAZE, channel-590
]

[romac]

Thanks for the report! Can you please re-run the command that failed with the --debug=rpc flag and post the logs here?

eg.

hermes --debug=rpc --config ./config.toml clear packets --chain uni-6 --channel channel-721 --port wasm.juno1uvflayc34fs9r3tzcausztks2058ecpt5kqtzf3sz5kwnu7fjtfqn7zmtj

[taitruong]

Thanks for the report! Can you please re-run the command that failed with the --debug=rpc flag and post the logs here?

eg.

hermes --debug=rpc --config ./config.toml clear packets --chain uni-6 --channel channel-721 --port wasm.juno1uvflayc34fs9r3tzcausztks2058ecpt5kqtzf3sz5kwnu7fjtfqn7zmtj

Here we go:

2023-10-13T20:27:57.281928Z  INFO ThreadId(01) running Hermes v1.6.0+1c1cf02
2023-10-13T20:27:57.283746Z DEBUG ThreadId(01) Outgoing request: {
  "jsonrpc": "2.0",
  "id": "de361d46-3b8b-4fb1-8d24-1046c7dced11",
  "method": "status",
  "params": null
}
2023-10-13T20:27:57.484373Z DEBUG ThreadId(01) Incoming response: {"jsonrpc":"2.0","id":"de361d46-3b8b-4fb1-8d24-1046c7dced11","result":{"node_info":{"protocol_version":{"p2p":"8","block":"11","app":"0"},"id":"0d194f5236a420147b05d9556ac0cf928c36e647","listen_addr":"65.109.23.114:11756","network":"pisco-1","version":"0.37.2","channels":"40202122233038606100","moniker":"hello-terra-testrelay","other":{"tx_index":"on","rpc_address":"tcp://0.0.0.0:11757"}},"sync_info":{"latest_block_hash":"2B6CD648B9355F4A7294779B49892618C0A35FCAEB01314A97737B608464BE24","latest_app_hash":"6B403B26E89CAA0893D18E496F3573AC027A960C4E66231D62B39E567EB3543E","latest_block_height":"7873519","latest_block_time":"2023-10-13T20:27:50.456001077Z","earliest_block_hash":"F9A088A259FC93FED2B053BEB5C097DEC0B3984D975D5E23C882795D9F2AB2E6","earliest_app_hash":"48D600F209AAA3EA64A18444CD99B3E24EA1E65F4DC1925055BA36C424A0724E","earliest_block_height":"6660741","earliest_block_time":"2023-07-25T13:58:54.797854271Z","catching_up":false},"validator_info":{"address":"99281AC70CA77998969E24A595363E81BCF6FFCE","pub_key":{"type":"tendermint/PubKeyEd25519","value":"L1HjhqkxoL+lVG3hgZSR2ktUoil3aJvyxpUNWHSDNkE="},"voting_power":"0"}}}
2023-10-13T20:27:57.487989Z DEBUG ThreadId(04) Outgoing request: {
  "jsonrpc": "2.0",
  "id": "1578544c-7f58-4721-b61a-a20b7c07a586",
  "method": "abci_query",
  "params": {
    "path": "store/ibc/key",
    "data": "6368616E6E656C456E64732F706F7274732F7761736D2E7465727261316C756879326E676A3779306C717933766A656B33786D37373770797770747A7A656130736335796D7534706D61666A726E30377335726C6D78742F6368616E6E656C732F6368616E6E656C2D333534",
    "prove": false
  }
}
2023-10-13T20:27:57.611200Z DEBUG ThreadId(04) Incoming response: {"jsonrpc":"2.0","id":"1578544c-7f58-4721-b61a-a20b7c07a586","result":{"response":{"code":0,"log":"","info":"","index":"0","key":"Y2hhbm5lbEVuZHMvcG9ydHMvd2FzbS50ZXJyYTFsdWh5Mm5najd5MGxxeTN2amVrM3htNzc3cHl3cHR6emVhMHNjNXltdTRwbWFmanJuMDdzNXJsbXh0L2NoYW5uZWxzL2NoYW5uZWwtMzU0","value":"CAMQARpUCkV3YXNtLnN0YXJzMTY1ZnI1ajNlYTBkazV5cjVzcjRoOHV3NzZ6NWZkMzRscG5xYzJ4ODB5YTl4a3g5dTBrM3FqcnFwenESC2NoYW5uZWwtNTkwIg5jb25uZWN0aW9uLTQwOSoIaWNzNzIxLTE=","proofOps":null,"height":"7873519","codespace":""}}}
2023-10-13T20:27:57.730925Z DEBUG ThreadId(04) Outgoing request: {
  "jsonrpc": "2.0",
  "id": "f48d17d7-3934-43aa-b27f-6f3270219aec",
  "method": "abci_query",
  "params": {
    "path": "store/ibc/key",
    "data": "636C69656E74732F30372D74656E6465726D696E742D3435322F636C69656E745374617465",
    "prove": false
  }
}
2023-10-13T20:27:57.789098Z DEBUG ThreadId(04) Incoming response: <html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

ERROR supervisor error: relayer error: RPC error to endpoint https://terra-testnet-rpc.polkachu.com/: serde parse error: expected value at line 1 column 1

[taitruong]

Seems like hermes start is doing to many requests. Can I change this?

[taitruong]

I guess rate limiter gets into account. Depending in what state hermes is ERROR link initialization failed during channel counterparty verification: failed during a query to chain 'pisco-1': RPC error to endpoint https://terra-testnet-rpc.polkachu.com/: serde parse error: expected value at line 1 column 1 also pops up:

2023-10-13T21:00:53.341733Z  INFO ThreadId(01) running Hermes v1.6.0+1c1cf02
2023-10-13T21:00:53.475585Z DEBUG ThreadId(01) Outgoing request: {
  "jsonrpc": "2.0",
  "id": "6eae7c17-c9b3-40d8-9b4e-9e7e977876cd",
  "method": "status",
  "params": null
}
2023-10-13T21:00:53.688969Z DEBUG ThreadId(01) Incoming response: {"jsonrpc":"2.0","id":"6eae7c17-c9b3-40d8-9b4e-9e7e977876cd","result":{"node_info":{"protocol_version":{"p2p":"8","block":"11","app":"0"},"id":"0d194f5236a420147b05d9556ac0cf928c36e647","listen_addr":"65.109.23.114:11756","network":"pisco-1","version":"0.37.2","channels":"40202122233038606100","moniker":"hello-terra-testrelay","other":{"tx_index":"on","rpc_address":"tcp://0.0.0.0:11757"}},"sync_info":{"latest_block_hash":"6F1941D76C211BCF05126C858C4078DAD527D48DBB6AADA36C25123E852BCAFE","latest_app_hash":"D7E6B3E73D400D35B15627EDD7C2066E355972C1358E9615B7A72500FBA9D585","latest_block_height":"7873863","latest_block_time":"2023-10-13T21:00:42.670667975Z","earliest_block_hash":"F9A088A259FC93FED2B053BEB5C097DEC0B3984D975D5E23C882795D9F2AB2E6","earliest_app_hash":"48D600F209AAA3EA64A18444CD99B3E24EA1E65F4DC1925055BA36C424A0724E","earliest_block_height":"6660741","earliest_block_time":"2023-07-25T13:58:54.797854271Z","catching_up":false},"validator_info":{"address":"99281AC70CA77998969E24A595363E81BCF6FFCE","pub_key":{"type":"tendermint/PubKeyEd25519","value":"L1HjhqkxoL+lVG3hgZSR2ktUoil3aJvyxpUNWHSDNkE="},"voting_power":"0"}}}
2023-10-13T21:00:54.473440Z DEBUG ThreadId(04) Outgoing request: {
  "jsonrpc": "2.0",
  "id": "7d842725-b8c5-4f7b-86b1-bc5cafe53626",
  "method": "abci_query",
  "params": {
    "path": "store/ibc/key",
    "data": "6368616E6E656C456E64732F706F7274732F7761736D2E7465727261316C756879326E676A3779306C717933766A656B33786D37373770797770747A7A656130736335796D7534706D61666A726E30377335726C6D78742F6368616E6E656C732F6368616E6E656C2D333534",
    "prove": false
  }
}
2023-10-13T21:00:54.594741Z DEBUG ThreadId(04) Incoming response: {"jsonrpc":"2.0","id":"7d842725-b8c5-4f7b-86b1-bc5cafe53626","result":{"response":{"code":0,"log":"","info":"","index":"0","key":"Y2hhbm5lbEVuZHMvcG9ydHMvd2FzbS50ZXJyYTFsdWh5Mm5najd5MGxxeTN2amVrM3htNzc3cHl3cHR6emVhMHNjNXltdTRwbWFmanJuMDdzNXJsbXh0L2NoYW5uZWxzL2NoYW5uZWwtMzU0","value":"CAMQARpUCkV3YXNtLnN0YXJzMTY1ZnI1ajNlYTBkazV5cjVzcjRoOHV3NzZ6NWZkMzRscG5xYzJ4ODB5YTl4a3g5dTBrM3FqcnFwenESC2NoYW5uZWwtNTkwIg5jb25uZWN0aW9uLTQwOSoIaWNzNzIxLTE=","proofOps":null,"height":"7873864","codespace":""}}}
2023-10-13T21:00:54.719752Z DEBUG ThreadId(04) Outgoing request: {
  "jsonrpc": "2.0",
  "id": "16f53224-729d-4d9d-9646-6b40b4cebc59",
  "method": "abci_query",
  "params": {
    "path": "store/ibc/key",
    "data": "636C69656E74732F30372D74656E6465726D696E742D3435322F636C69656E745374617465",
    "prove": false
  }
}
2023-10-13T21:00:54.776200Z DEBUG ThreadId(04) Incoming response: {"jsonrpc":"2.0","id":"16f53224-729d-4d9d-9646-6b40b4cebc59","result":{"response":{"code":0,"log":"","info":"","index":"0","key":"Y2xpZW50cy8wNy10ZW5kZXJtaW50LTQ1Mi9jbGllbnRTdGF0ZQ==","value":"CisvaWJjLmxpZ2h0Y2xpZW50cy50ZW5kZXJtaW50LnYxLkNsaWVudFN0YXRlEoEBCgllbGdhZmFyLTESBAgBEAMaBAiAnDEiBAiA6kkqAghzMgA6BwgBEP+ltgNCGQoJCAEYASABKgEAEgwKAgABECEYBCAMMAFCGQoJCAEYASABKgEAEgwKAgABECAYASABMAFKB3VwZ3JhZGVKEHVwZ3JhZGVkSUJDU3RhdGVQAVgB","proofOps":null,"height":"7873864","codespace":""}}}
2023-10-13T21:00:54.778287Z DEBUG ThreadId(01) Outgoing request: {
  "jsonrpc": "2.0",
  "id": "0d691d1c-9274-442d-93a7-5c892046b7b3",
  "method": "status",
  "params": null
}
2023-10-13T21:00:54.938875Z DEBUG ThreadId(01) Incoming response: {"jsonrpc":"2.0","id":"0d691d1c-9274-442d-93a7-5c892046b7b3","result":{"node_info":{"protocol_version":{"p2p":"8","block":"11","app":"0"},"id":"89736cec483dd94ad946bb7325eeac9a1528f8e7","listen_addr":"tcp://0.0.0.0:26656","network":"elgafar-1","version":"0.34.29","channels":"40202122233038606100","moniker":"node1","other":{"tx_index":"on","rpc_address":"tcp://0.0.0.0:26657"}},"sync_info":{"latest_block_hash":"DA1BA44F24A288C7AF1A542C1341AFE534296026FF24D231EA07F1CB86B00AD7","latest_app_hash":"049FA764DA37CF3E7CF687EF9E4803E76C29BDFEF288A043595748CE21A25FCF","latest_block_height":"7181141","latest_block_time":"2023-10-13T21:00:47.873739366Z","earliest_block_hash":"3AB6080592AE56BDA8EE56123894F09D913EB2FFCD1232839462971C8FFE8B2F","earliest_app_hash":"E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855","earliest_block_height":"1","earliest_block_time":"2022-07-06T18:44:13.502116971Z","catching_up":false},"validator_info":{"address":"CBC8C35D0E967D267D50C745A74BE75F6BDFCDDF","pub_key":{"type":"tendermint/PubKeyEd25519","value":"TYVkCMsi+nGR9i8Hyimz8lb0tIQRZBWFNnUfF20ufgw="},"voting_power":"50125017"}}}
2023-10-13T21:00:54.940853Z DEBUG ThreadId(04) Outgoing request: {
  "jsonrpc": "2.0",
  "id": "a1dd87a9-dff8-46a2-8cc1-3a924b93e794",
  "method": "abci_query",
  "params": {
    "path": "store/ibc/key",
    "data": "6368616E6E656C456E64732F706F7274732F7761736D2E7465727261316C756879326E676A3779306C717933766A656B33786D37373770797770747A7A656130736335796D7534706D61666A726E30377335726C6D78742F6368616E6E656C732F6368616E6E656C2D333534",
    "prove": false
  }
}
2023-10-13T21:00:55.063477Z DEBUG ThreadId(04) Incoming response: {"jsonrpc":"2.0","id":"a1dd87a9-dff8-46a2-8cc1-3a924b93e794","result":{"response":{"code":0,"log":"","info":"","index":"0","key":"Y2hhbm5lbEVuZHMvcG9ydHMvd2FzbS50ZXJyYTFsdWh5Mm5najd5MGxxeTN2amVrM3htNzc3cHl3cHR6emVhMHNjNXltdTRwbWFmanJuMDdzNXJsbXh0L2NoYW5uZWxzL2NoYW5uZWwtMzU0","value":"CAMQARpUCkV3YXNtLnN0YXJzMTY1ZnI1ajNlYTBkazV5cjVzcjRoOHV3NzZ6NWZkMzRscG5xYzJ4ODB5YTl4a3g5dTBrM3FqcnFwenESC2NoYW5uZWwtNTkwIg5jb25uZWN0aW9uLTQwOSoIaWNzNzIxLTE=","proofOps":null,"height":"7873864","codespace":""}}}
2023-10-13T21:00:55.063773Z DEBUG ThreadId(08) Outgoing request: {
  "jsonrpc": "2.0",
  "id": "2ee13776-a48c-4852-84a3-521f55fec070",
  "method": "abci_query",
  "params": {
    "path": "store/ibc/key",
    "data": "6368616E6E656C456E64732F706F7274732F7761736D2E73746172733136356672356A33656130646B357972357372346838757737367A35666433346C706E716332783830796139786B783975306B33716A7271707A712F6368616E6E656C732F6368616E6E656C2D353930",
    "prove": false
  }
}
2023-10-13T21:00:55.198893Z DEBUG ThreadId(08) Incoming response: {"jsonrpc":"2.0","id":"2ee13776-a48c-4852-84a3-521f55fec070","result":{"response":{"code":0,"log":"","info":"","index":"0","key":"Y2hhbm5lbEVuZHMvcG9ydHMvd2FzbS5zdGFyczE2NWZyNWozZWEwZGs1eXI1c3I0aDh1dzc2ejVmZDM0bHBucWMyeDgweWE5eGt4OXUwazNxanJxcHpxL2NoYW5uZWxzL2NoYW5uZWwtNTkw","value":"CAMQARpUCkV3YXNtLnRlcnJhMWx1aHkybmdqN3kwbHF5M3ZqZWszeG03NzdweXdwdHp6ZWEwc2M1eW11NHBtYWZqcm4wN3M1cmxteHQSC2NoYW5uZWwtMzU0Ig5jb25uZWN0aW9uLTU5NyoIaWNzNzIxLTE=","proofOps":null,"height":"7181141","codespace":""}}}
2023-10-13T21:00:55.318743Z DEBUG ThreadId(08) Outgoing request: {
  "jsonrpc": "2.0",
  "id": "d6b2730c-1e18-4c09-83f7-8b2b5e5d18ec",
  "method": "abci_query",
  "params": {
    "path": "store/ibc/key",
    "data": "6368616E6E656C456E64732F706F7274732F7761736D2E73746172733136356672356A33656130646B357972357372346838757737367A35666433346C706E716332783830796139786B783975306B33716A7271707A712F6368616E6E656C732F6368616E6E656C2D353930",
    "prove": false
  }
}
2023-10-13T21:00:55.458099Z DEBUG ThreadId(08) Incoming response: {"jsonrpc":"2.0","id":"d6b2730c-1e18-4c09-83f7-8b2b5e5d18ec","result":{"response":{"code":0,"log":"","info":"","index":"0","key":"Y2hhbm5lbEVuZHMvcG9ydHMvd2FzbS5zdGFyczE2NWZyNWozZWEwZGs1eXI1c3I0aDh1dzc2ejVmZDM0bHBucWMyeDgweWE5eGt4OXUwazNxanJxcHpxL2NoYW5uZWxzL2NoYW5uZWwtNTkw","value":"CAMQARpUCkV3YXNtLnRlcnJhMWx1aHkybmdqN3kwbHF5M3ZqZWszeG03NzdweXdwdHp6ZWEwc2M1eW11NHBtYWZqcm4wN3M1cmxteHQSC2NoYW5uZWwtMzU0Ig5jb25uZWN0aW9uLTU5NyoIaWNzNzIxLTE=","proofOps":null,"height":"7181141","codespace":""}}}
2023-10-13T21:00:55.458358Z DEBUG ThreadId(04) Outgoing request: {
  "jsonrpc": "2.0",
  "id": "00fd73df-6b00-482c-b94e-5325947cd4ee",
  "method": "abci_query",
  "params": {
    "path": "store/ibc/key",
    "data": "6368616E6E656C456E64732F706F7274732F7761736D2E7465727261316C756879326E676A3779306C717933766A656B33786D37373770797770747A7A656130736335796D7534706D61666A726E30377335726C6D78742F6368616E6E656C732F6368616E6E656C2D333534",
    "prove": false
  }
}
2023-10-13T21:00:55.584325Z DEBUG ThreadId(04) Incoming response: <html>
<head><title>429 Too Many Requests</title></head>
<body>
<center><h1>429 Too Many Requests</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

ERROR link initialization failed during channel counterparty verification: failed during a query to chain 'pisco-1': RPC error to endpoint https://terra-testnet-rpc.polkachu.com/: serde parse error: expected value at line 1 column 1

[taitruong]

Sometime I also get an ERROR channel wasm.terra1luhy2ngj7y0lqy3vjek3xm777pywptzzea0sc5ymu4pmafjrn07s5rlmxt/channel-354 does not exist on chain pisco-1: RPC error to endpoint https://terra-testnet-rpc.polkachu.com/: serde parse error: expected value at line 1 column 1

Which is quite misleading, since channel does exist!

[romac]

Sometime I also get an ERROR channel wasm.terra1luhy2ngj7y0lqy3vjek3xm777pywptzzea0sc5ymu4pmafjrn07s5rlmxt/channel-354 does not exist on chain pisco-1: RPC error to endpoint https://terra-testnet-rpc.polkachu.com/: serde parse error: expected value at line 1 column 1

Which is quite misleading, since channel does exist!

Agreed, that's a very misleading error message, we'll fix it.

[romac]

Seems like hermes start is doing to many requests. Can I change this?

Are you connecting to a public node? If so then there is not much that Hermes can do. I mean we could rate limit the requests that Hermes performs but that would totally kill performance. In general, one cannot expect to be able to operate a relayer using a public node, and should instead use a private, appropriately configured, node.

[taitruong]

Seems like hermes start is doing to many requests. Can I change this?

Are you connecting to a public node? If so then there is not much that Hermes can do. I mean we could rate limit the requests that Hermes performs but that would totally kill performance. In general, one cannot expect to be able to operate a relayer using a public node, and should instead use a private, appropriately configured, node.

Yes, connecting to a public node - so not locally. It would be great for having a rate limit flag. Imo, better having a lower performance - than not relaying anything at all.