Intermittent issue Recipient not found for key

[bipeen-kumar]

System information

We are running Quorum on AKS cluster with below configuration-

Org A on Cluster1
Quorum

• 3 pods sharing same nodekey and enode
• Version: 22.4.0
• Flags:
  RPC_APIS=admin,db,eth,debug,miner,net,shh,txpool,personal,web3,quorum,quorumExtension,istanbul
args=" --gcmode archive --istanbul.blockperiod 5 --syncmode full "
usr/local/bin/geth --datadir $QUORUM_DATA_DIR $args --permissioned --keystore /etc/quorum/qdata/keystore/ --nodekey /etc/quorum/qdata/nodekey --nodiscover --nat=none --verbosity 9 --networkid $NETWORK_ID --emitcheckpoints --rpc --rpcaddr 0.0.0.0 --rpcvhosts=* --rpcport 8545 --rpcapi $RPC_APIS --wsapi $RPC_APIS --wsport 8546 --port 30303

Tessera

• 3 replicas of tessera
• Version: 22.1.1

Same configuration is done Org B on Cluster2

Expected behaviour

All the private transactions should be successful.

Actual behaviour

We have 5 times retry on the client side for this error and Let's consider if we submit 5 transactions-
A couple of transactions go through without error, a few transactions go through after 2 or 3 reties from the client side but few transactions fail even after 5 retries with error "Recipient not found for key: counterparty-tessera-pub-key"

In the tessera logs we don't see any error.

In the quorum logs we do see below warnings/trace/debug logs
WARN [08-16|15:48:34.962] Served eth_sendRawPrivateTransaction conn=10.8.0.142:32890 reqid=184t=80.291437ms err="404 status: Recipient not found for key: +HchS+uE+z9nyHiw="

DEBUG[08-16|16:00:38.512] RPC connection read error err=EOF

TRACE[08-16|15:48:35.169] Failed RLPx handshakeaddr=10.8.0.129:57813conn=inbounderr="read tcp 10.8.0.11:30303->10.8.0.129:57813: i/o timeout"

[baptiste-b-pegasys]

Hello,
Is it specific to one specific node that is not able to connect to tessera corresponding instance?
What is the log of this tessera instance?
Are you using https://github.com/ConsenSys/quorum-kubernetes/ and raised the issue here?

[bipeen-kumar]

@baptiste-b-pegasys The error is coming up randomly from all the 3 quorum nodes, it is not specific to one node and there are no errors in the tessera pod.

We are not entirely using https://github.com/ConsenSys/quorum-kubernetes/ but the setup is somewhat similar.

[baptiste-b-pegasys]

are those 3 nodes are validators or simple rpc nodes?

[bipeen-kumar]

We have 4 validators and 3 quorum transaction nodes and 3 replicas of tessera.

[bipeen-kumar]

We also observed one thing after restarting the quorum pods the issue is no longer coming up.

[bipeen-kumar]

@baptiste-b-pegasys Were you able to find anything helpful for this issue? or could you share your thoughts on the reason for this issue?

This issue is now coming up in multiple environments.

[baptiste-b-pegasys]

Hello, sorry for the late response.
I can try to reproduce this. Do you have some config files or minimal environment in order to reproduce it.

My thoughts is that the 3 nodes can't be validator just a RPC node, but as I understood from your previous answer, it is the case.

Then it may worth to check the process used by the function called in concurrent mode: are these transactions come from the same originator (from)? is the nonce unique for every tx?

thank you

[bipeen-kumar]

@baptiste-b-pegasys Yes, we have some environments where we can reproduce the issue.
Are these transactions come from the same originator (from)? - Yes, we have transactions from the same originator.
Is the nonce unique for every tx?- Yes (We also ensure the transaction always gets posted in sequence)

[antonydenyer]

We think this has been fixed in Consensys/tessera#1489 and released https://github.com/ConsenSys/tessera/releases/tag/tessera-22.10.0