publishBlockV2 fails gossip validation for valid block

[nflaig]

Describe the bug

I am testing Nimbus BN with Lodestar VC on Kurtosis. The previous issue (#6176) is now fixed on unstable branch (c5f04dd) but there is another issue during block publishing which only happens with Nimbus from unstable branch, but works on stable branch.

This is the error logged on the Nimbus BN

WRN 2024-04-16 12:01:56.939+00:00 Block failed validation                    topics="beacval" blockRoot=00000000 blck="(slot: 1, proposer_index: 1147, parent_root: \"515bb412\", state_root: \"145b6448\", eth1data: (deposit_root: d70a234731285c6804c2a4f56711ddb8c82c99740f207854891028af34e27e5e, deposit_count: 0, block_hash: 6903b6723e1921a599446c55bbb140dac226f150d5ba737bd94456171c8b3ad9), graffiti: \"5-geth-nimbus-lodestar\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 0, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 0, block_number: 1, block_hash: \"0x8e132997d62f55007d8a1d67fe7eca60c5f2f738dbe47e1790f582db2cf25efe\", parent_hash: \"0x6903b6723e1921a599446c55bbb140dac226f150d5ba737bd94456171c8b3ad9\", fee_recipient: \"0x8943545177806ed17b9f23f0a21ee5948ecaa776\", bls_to_execution_changes_len: 0, blob_kzg_commitments_len: 0)" signature=aefb60f2 error="(Reject, \"BeaconBlock: Invalid proposer signature\")"

From the looks of it, it seems like the block is failing gossip validation, the publish request done by Lodestar looks like this

Apr 16 12:01:56.928 2024 REQUEST /eth/v2/beacon/blocks?broadcast_validation=gossip
�[38;5;6m{�[39m
�[38;5;6m  "message": {�[39m
�[38;5;6m    "slot": "1",�[39m
�[38;5;6m    "proposer_index": "1147",�[39m
�[38;5;6m    "parent_root": "0x515bb41259ab0902ad3e5ec658327cd25f387dfd5cf7763f7cd593060e213e7f",�[39m
�[38;5;6m    "state_root": "0x145b6448f76dd454df1b339ae4731e6e30ce73b2c8ea4a008f290899de2c339a",�[39m
�[38;5;6m    "body": {�[39m
�[38;5;6m      "randao_reveal": "0x87b1cce981a31d800402667eee9c7de84de72c0a01ffbe8744106aed6c50660faa5faed1e5846a08207202784bb28105054d9425621e8b56db41dde74d2b6ab775f4dac84bb43593db34f65570ba296b025a86da859dd9a5097c4a083b0932c5",�[39m
�[38;5;6m      "eth1_data": {�[39m
�[38;5;6m        "deposit_root": "0xd70a234731285c6804c2a4f56711ddb8c82c99740f207854891028af34e27e5e",�[39m
�[38;5;6m        "deposit_count": "0",�[39m
�[38;5;6m        "block_hash": "0x6903b6723e1921a599446c55bbb140dac226f150d5ba737bd94456171c8b3ad9"�[39m
�[38;5;6m      },�[39m
�[38;5;6m      "graffiti": "0x352d676574682d6e696d6275732d6c6f64657374617200000000000000000000",�[39m
�[38;5;6m      "proposer_slashings": [],�[39m
�[38;5;6m      "attester_slashings": [],�[39m
�[38;5;6m      "attestations": [],�[39m
�[38;5;6m      "deposits": [],�[39m
�[38;5;6m      "voluntary_exits": [],�[39m
�[38;5;6m      "sync_aggregate": {�[39m
�[38;5;6m        "sync_committee_bits": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",�[39m
�[38;5;6m        "sync_committee_signature": "0xc00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"�[39m
�[38;5;6m      },�[39m
�[38;5;6m      "execution_payload": {�[39m
�[38;5;6m        "parent_hash": "0x6903b6723e1921a599446c55bbb140dac226f150d5ba737bd94456171c8b3ad9",�[39m
�[38;5;6m        "fee_recipient": "0x8943545177806ED17B9F23F0a21ee5948eCaa776",�[39m
�[38;5;6m        "state_root": "0xaa65ab8098cfddd2a7e893480584e2a331ea7be5a91afede0bce3b8e4515e643",�[39m
�[38;5;6m        "receipts_root": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",�[39m
�[38;5;6m        "logs_bloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",�[39m
�[38;5;6m        "prev_randao": "0x6903b6723e1921a599446c55bbb140dac226f150d5ba737bd94456171c8b3ad9",�[39m
�[38;5;6m        "block_number": "1",�[39m
�[38;5;6m        "gas_limit": "25024413",�[39m
�[38;5;6m        "gas_used": "0",�[39m
�[38;5;6m        "timestamp": "1713268916",�[39m
�[38;5;6m        "extra_data": "0xd883010d0e846765746888676f312e32312e37856c696e7578",�[39m
�[38;5;6m        "base_fee_per_gas": "875000000",�[39m
�[38;5;6m        "block_hash": "0x8e132997d62f55007d8a1d67fe7eca60c5f2f738dbe47e1790f582db2cf25efe",�[39m
�[38;5;6m        "transactions": [],�[39m
�[38;5;6m        "withdrawals": []�[39m
�[38;5;6m      },�[39m
�[38;5;6m      "bls_to_execution_changes": []�[39m
�[38;5;6m    }�[39m
�[38;5;6m  },�[39m
�[38;5;6m  "signature": "0xaefb60f20fce54c1c9a4669f2a5d7b9fda279b602b60221d57befcb2d78358232542494ab89530a21d72229784e1b3d4009f0f04967a0e19ead7cb45a9f38739cc17cdcdd148701d7160886816432e183339ecc715f2d4203c8cf19d9366e800"�[39m
�[38;5;6m}�[39m

And the response from Nimbus BN

Apr 16 12:01:56.939 2024 RESPONSE (status 503 Service Unavailable) 
�[38;5;2m{�[39m
�[38;5;2m  "code": 503,�[39m
�[38;5;2m  "message": "Beacon node is currently syncing and not serving request on that endpoint",�[39m
�[38;5;2m  "stacktraces": [�[39m
�[38;5;2m    "BeaconBlock: Invalid proposer signature"�[39m
�[38;5;2m  ]�[39m
�[38;5;2m}�[39m

To Reproduce
Steps to reproduce the behavior:

Kurtosis config file

participants:
# ...
  - el_type: geth
    el_image: ethereum/client-go:stable
    cl_type: nimbus
    cl_image: ethpandaops/nimbus-eth2:unstable-c5f04dd
    vc_type: lodestar
    vc_image: chainsafe/lodestar:next
    # vc_image: nflaig/lodestar:use-publish-v1 <-- using publish v1 api solves the issue
    count: 1
# ...

Additional context

The issue seems to be related to publishBlockV2 only, forcing Lodestar VC to use publishBlock (v1) resolves the issue.

[cheatfate]

This issue should be fixed in unstable.