L2 synchronization is delayed for several hours due to OP-NODE's failure to parse L1 blocks

[junha-ahn]

Situation:

L2 synchronization is delayed for several hours due to OP-NODE's failure to parse L1 blocks.

Then, the synchronization is suddenly completed to the latest block.

Environment:

Manta Node Configure

• L1: Ethereum network
• L2: Manta network (using OP-Node, OP-Geth, Celestia)

docker-compose.yaml:

celestia:
   ...
op-geth:
   ... 
op-node:
    image: ".../op-node:celestia-3.0.0"
    command: >
      op-node
      --l1="${L1_RPC_URL}"
      --l2=http://l2:8551
      --l2.jwt-secret=/config/test-jwt-secret.txt 
      ...

      --p2p.listen.ip=0.0.0.0
      --p2p.listen.tcp=9003
      --p2p.listen.udp=9003
      --p2p.static=/ip4/35.82.210.70/tcp/9003/p2p/16Uiu2HAmL4fvgBQi5jcuiEYDaNcg4hpGqCmyAv4DZuSM8f2USYwQ

Repeated Error Messages:

During the delay, the following error messages are continuously displayed:

manta-pacific-op-node-1  | t=2024-02-15T02:01:07+0000 lvl=warn msg="Derivation process temporary error"     attempts=1665  err="temp: failed to fetch receipts of L1 block 0x855a9a2ec4a098bf17d7b7d7887151cf8e65425e7364c85afbd4741d8f0ac064:19228780 for L1 sysCfg update: receipt 0 has unexpected tx index 4"
manta-pacific-op-node-1  | t=2024-02-15T02:01:08+0000 lvl=info msg="attempting connection"                  peer=16Uiu2HAmL4fvgBQi5jcuiEYDaNcg4hpGqCmyAv4DZuSM8f2USYwQ

Issue 1: Repeated "attempting connection"

Currently, OP-Node is failing to connect to other P2P peers (includes static p2p peer), as indicated by the continuous "attempting connection" error message.

• static p2p peer: 16Uiu2HAmL4fvgBQi5jcuiEYDaNcg4hpGqCmyAv4DZuSM8f2USYwQ

$ curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"opp2p_peers","params":[true],"id":1}' http://localhost:7545
{"jsonrpc":"2.0","id":1,"result":{"totalConnected":0,"peers":{},"bannedPeers":[],"bannedIPS":[],"bannedSubnets":[]}}

$ nc 35.82.210.70 9003
Connection to 35.82.210.70 port 9003 [tcp/*] succeeded!

The firewall is confirmed to be not an issue.

The following error message appears when op-node starts:

manta-pacific-op-node-1  | t=2024-02-15T07:18:03+0000 lvl=warn msg="error dialing static peer"         peer=16Uiu2HAmL4fvgBQi5jcuiEYDaNcg4hpGqCmyAv4DZuSM8f2USYwQ err="failed to dial 16Uiu2HAmL4fvgBQi5jcuiEYDaNcg4hpGqCmyAv4DZuSM8f2USYwQ:\n  * [/ip4/35.82.210.70/tcp/9003] failed to negotiate security protocol: peer id mismatch: expected 16Uiu2HAmL4fvgBQi5jcuiEYDaNcg4hpGqCmyAv4DZuSM8f2USYwQ, but remote key matches 16Uiu......NfcH"

After this message, "attempting connection" is repeated.

Could you please provide any insights or potential causes for the P2P connection failure?

Issue 2: Repeated "failed to fetch receipts"

$ curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"optimism_syncStatus","params":[],"id":1}' http://localhost:7545

# summary
- current_l1: 19231378 / 05:55:11 AM
- current_l1_finalized: 19231571 / 06:33:59 AM
- unsafe_l2: 1375229 / 5:50:49 AM

The above message was printed at 10 AM, which means that the L2 block synchronization was delayed by about 4 hours.

L1 RPC itself is working fine. (sync well)

L1 RPC is currently composed of multiple eth-geth nodes through a load balancer. (The height of each synchronization block may vary slightly)

However, why is it that L1 blocks were not parsed for several hours and then suddenly resolved all at once?

Summary

1. P2P connection issue
2. L1 block parsing issue (or etc)

I think the issue of delay in synchronization depends on issue 2.