Lodestar CL < > Prysm VC incompatibility

[barnabasbusa]

Describe the bug

We are in the process to test cross beacon <> validator client compatibility, and found a bug when testing lodestar CL with prysm VC.

Prysm reports:

time="2024-04-05 11:37:45" level=info msg="Syncing with beacon node to align on chain genesis info" prefix=client
time="2024-04-05 11:37:45" level=warning msg="Could not determine if beacon chain started" error="could not receive ChainStart from stream: failed to get genesis data: genesis data is nil: could not connect" prefix=client```

Lodestar reports:

Apr-05 11:53:30.002[chain]         verbose: Clock slot slot=7
Apr-05 11:53:32.422[network]         debug: Req  received method=ping, client=Teku, peer=16...36zo6L, requestId=38
Apr-05 11:53:32.423[network]       verbose: Resp done method=ping, client=Teku, peer=16...36zo6L, requestId=38
Apr-05 11:53:33.845[network]         debug: peerManager heartbeat result peersToDisconnect=0, peersToConnect=51, attnetQueries=66, syncnetQueries=0
Apr-05 11:53:34.764[rest]            debug: Req req-u 172.16.0.30 getGenesis
Apr-05 11:53:34.765[rest]            debug: Exec req-u 172.16.0.30 getGenesis
Apr-05 11:53:34.766[rest]            debug: Res req-u getGenesis - 200
Apr-05 11:53:35.069[network]         debug: Req  received method=ping, client=Lodestar, peer=16...vqEsea, requestId=39
Apr-05 11:53:35.070[network]       verbose: Resp done method=ping, client=Lodestar, peer=16...vqEsea, requestId=39
Apr-05 11:53:36.001[]                 info: Synced - slot: 7 - head: (slot -3) 0x962f…78f0 - exec-block: valid(2 0x58ee…) - finalized: 0x0000…0000:0 - peers: 4

Snooper between cl <> vc reports:

Apr  5 11:54:34.759 2024 REQUEST /eth/v1/beacon/genesis
null

Apr  5 11:54:34.762 2024 RESPONSE (status 200 OK)
{
  "data": {
    "genesis_validators_root": "0x0ac76415092e45ba9068ffc47402b95f95cb0b1f1f5ab818b6ba5ddbe9e22c9e",
    "genesis_time": "1712317926",
    "genesis_fork_version": "0x10000038"
  }
}

It seems like prysm is stuck on requesting genesis state, and stops there.

Expected behavior

I would expect all client combinations would work.

Steps to reproduce

config.yaml:

participants:
  - cl_type: lodestar
    cl_image: chainsafe/lodestar:next
    vc_type: prysm
additional_services:
  - dora
snooper_enabled: true
global_log_level: debug

kurtosis run github.com/kurtosis-tech/ethereum-package --args-file config.yaml

Additional context

Current BN <> VC Compatibility list tracker

Operating system

Linux

Lodestar version or commit hash

Version: v1.17.0/898cd90
cc: @pk910

[nflaig]

It seems like prysm is stuck on requesting genesis state, and stops there.

This sounds similar to what is reported in prysmaticlabs/prysm#13852, prysmaticlabs/prysm#13851 and status-im/nimbus-eth2#6175.