Light client - not receiving all block headers over gossipsub

[Eligioo]

New issue checklist

• I have read all of the README
• I have searched existing issues and this is not a duplicate.

General information

• Library version(s): 0.20
• Reproducible in the testnet? (Yes/No): yes

Bug report

With version 0.20 we are currently observing that light clients don't receive all block header messages over gossipsub. This prevents having a smooth way of accepting newly produced blocks as the block_queue has to request missing blocks constantly. The client basically isn't aware the chain has moved on until it receives a new gossipsub so it can request missing blocks again.

Expected behavior

A light client to receive all the blocker header messages from the its connected peers.

Actual behavior

Not receiving all messages

Steps to reproduce

Set the sync_mode to light in the client.toml and run the client.

Crash log? Screenshots? Videos? Sample project?

This log contains only log messages of received gossipsub messages

• note that there are gaps
• gossipsubs for the same block is received multiple times by different peers
• old, non-relevant gossipsubs are received when the head of the client is higher

missing-gossipsubs-filtered.log

2024-03-05T09:21:13.670000128Z DEBUG block_queue | Received block via gossipsub block=#17869370:MI:242edbb2bd peer_id=12D3KooWF4pZSd4wPcTqwhWVCBkV3T4qWdzdQSKpyHE342c64xeA
2024-03-05T09:21:21.659000064Z DEBUG block_queue | Received block via gossipsub block=#17869378:MI:df283ae507 peer_id=12D3KooWF4pZSd4wPcTqwhWVCBkV3T4qWdzdQSKpyHE342c64xeA
2024-03-05T09:21:22.380000000Z DEBUG block_queue | Received block via gossipsub block=#17869378:MI:df283ae507 peer_id=12D3KooWAjftHreEDsNMK3qPBSe63LZcQorv6q8H3FR7CWg57CxV
2024-03-05T09:21:23.656999936Z DEBUG block_queue | Received block via gossipsub block=#17869380:MI:62d7fee408 peer_id=12D3KooWF4pZSd4wPcTqwhWVCBkV3T4qWdzdQSKpyHE342c64xeA
2024-03-05T09:21:31.662000128Z DEBUG block_queue | Received block via gossipsub block=#17869388:MI:32a75bb061 peer_id=12D3KooWF4pZSd4wPcTqwhWVCBkV3T4qWdzdQSKpyHE342c64xeA
2024-03-05T09:21:32.361999872Z DEBUG block_queue | Received block via gossipsub block=#17869388:MI:32a75bb061 peer_id=12D3KooWAjftHreEDsNMK3qPBSe63LZcQorv6q8H3FR7CWg57CxV
2024-03-05T09:21:32.656999936Z DEBUG block_queue | Received block via gossipsub block=#17869389:MI:4efd2076c9 peer_id=12D3KooWF4pZSd4wPcTqwhWVCBkV3T4qWdzdQSKpyHE342c64xeA
2024-03-05T09:21:41.656000000Z DEBUG block_queue | Received block via gossipsub block=#17869398:MI:901c17ad0d peer_id=12D3KooWF4pZSd4wPcTqwhWVCBkV3T4qWdzdQSKpyHE342c64xeA
2024-03-05T09:21:42.379000064Z DEBUG block_queue | Received block via gossipsub block=#17869398:MI:901c17ad0d peer_id=12D3KooWAjftHreEDsNMK3qPBSe63LZcQorv6q8H3FR7CWg57CxV
2024-03-05T09:21:42.648999936Z DEBUG block_queue | Received block via gossipsub block=#17869399:MI:514cdbb604 peer_id=12D3KooWF4pZSd4wPcTqwhWVCBkV3T4qWdzdQSKpyHE342c64xeA
2024-03-05T09:21:47.651000064Z DEBUG block_queue | Received block via gossipsub block=#17869404:MI:edf8016961 peer_id=12D3KooWF4pZSd4wPcTqwhWVCBkV3T4qWdzdQSKpyHE342c64xeA

[sisou]

Potential partial explanation:

I just tested @nibhar's proof of concept branch nibhar/test/block-header-topic (which actively rebroadcasts full blocks on the separate header topic) on my full node and set it as the seed in my light client's config, and it works, which seems to confirm the theory that nodes don't forward received headers on the header pubsub topic when not subscribed themselves (which full nodes aren't), and light clients only get the headers from the producing validators they are connected to, leading to gaps when not connected to the validator that produced a block.

This was less noticeable before v0.19.0, when light clients connected to around 12 peers, so were connected to most if not all validators, so received most or all blocks directly from their sources.

With the desired peer count change in v0.19.0, being connected to less validators meant receiving less updates on the header topic, making it much more noticeable.