Light client - rejects block on live sync push event

[Eligioo]

New issue checklist

• I have read all of the README
• I have searched existing issues and this is not a duplicate.

General information

• Library version(s): 0.20
• Reproducible in the testnet? (Yes/No): Yes

Bug report

It has been observed that running a light client for some time, causes the node to reject a block at some point. This prevents the node from accepting new blocks and thus falls behind. It is noteworthy that all the rejected blocks are macro blocks so far.

The node doesn't lose consensus as it keeps receiving gossipsubs. Eventually the block queue notices that the gossipsubs are to too far ahead (head_height + window_ahead_max) and starts a macro sync. Which brings the node back to the actual head of the chain and can from there on accept new blocks properly. Until the node rejects a block again and then this cycle repeats itself.

Although the blocks are rejected, they actually end up on the blockchain. Here are some example hashes which can be found in the attached log file:

d9da6da5af60ec28c25e36f3d56e31f51121e0818ffc8aa2ca0bf82e9128d6b4
f2b1aca4f451d61c87e8df38ca8a986c512896db2d8ef0f6037a5dc2d53c298f
dd317a6905bfefc758c8a6108e1d66f7f4e58c8c16b1741f251e51a62dcad284

Expected behavior

Be able to push and accept newly received blocks

Actual behavior

Rejecting some blocks.
Reason: Head push operation failed because of Invalid block: Missing body

Steps to reproduce

Set the sync_mode to light in the client.toml and run the client.

Crash log? Screenshots? Videos? Sample project?

Search the log file for Rejected block
light-client-rejects-live-push-event.log

[sisou]

Unfortunately no reason is given in the WARN message. Could you enable TRACE logs and see if any reason is logged?

[Eligioo]

Head push operation failed because of Invalid block: Missing body

[sisou]

Interesting! I learned from the test branch linked in #2288 (comment) that macro blocks always need to be gossiped with their body. So some node somewhere is sending them without body, and the light node cannot disregard the message and wait for another one. Or maybe that's not how gossipsub works 🤷

[Eligioo]

@sisou those rejections always come from peer with ID 12D3KooWJMppaYVt6JuMxBUuTXcpvNrDDiyJZfGyP7UQvY8udUKD, which is your node that is checked out on the nibhar/test/block-header-topic branch.

[sisou]

Haha, oh shit, really? I thought this was an older issue. How I read the code in that branch, that shouldn't be happening though. @nibhar

[Eligioo]

What is happening here is that in the nibhar/test/block-header-topic reads the block without a body (self.blockchain.read().get_block(&block_hash, false)). This causes the rebroadcast of a macro block to not contain a body.

[sisou]

Fixed in aed4c43 and redeployed.

[hrxi]

Still seems like a problem that you can DoS a light client like this.