Sync problems

[maestroi]

New issue checklist

• I have read all of the README
• I have searched existing issues and this is not a duplicate.

General information

• Library version(s): 0.20.3
• Browser version(s): --
• Devices/Simulators/Machine affected: VPS
• Reproducible in the testnet? (Yes/No): ?
• Related issues:

Bug report

Expected behavior

Get consensus and producing blocks

Actual behavior

The node never gets to consensus state, even after removing full ledger it keeps getting stuck on several heights or never reaches consensus, I had 1 particual crash where it was unrecoverable
https://pastebin.com/gmKhCFrE

Steps to reproduce

Hard to say just have new validator and sometimes never get to tip of chain.

Crash log? Screenshots? Videos? Sample project?

https://pastebin.com/gmKhCFrE

Question or Feature Request

Seems to be bunch sync related issues, woould be good to prioritize it to make it more stable.

[redmaner]

Running into similar issues with a fresh full node on v0.20.3 it is at the head of the chain, but consensus is not established.

Mar 25 18:43:44 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:43:44.104687979Z INFO  nimiq_client         | Consensus: lost - Head: #19315161:MI:393f49317f - Peers: 20 consensus_established=false block_number=19315161 num_peers=20
Mar 25 18:43:54 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:43:54.104267925Z INFO  nimiq_client         | Consensus: lost - Head: #19315171:MI:3060ea76e7 - Peers: 20 consensus_established=false block_number=19315171 num_peers=20
Mar 25 18:43:54 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:43:54.184999578Z INFO  sync                 | Requesting zkp from peer peer_id=12D3KooWKmduaetweNxmzNQyqwYVSSXitwkPyDFHgehNHvJM5TMZ
Mar 25 18:44:04 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:04.104068695Z INFO  nimiq_client         | Consensus: lost - Head: #19315181:MI:7595177789 - Peers: 21 consensus_established=false block_number=19315181 num_peers=21
Mar 25 18:44:05 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:05.386286865Z WARN  behaviour            | Rejected message not in cache b5e8ed94e8d7579860298c85c45a837a6dc1288b3afa374f51ca0a318b886196
Mar 25 18:44:07 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:07.909054335Z INFO  consensus            | Catching up to tip of the chain (now at #19315185, 0 blocks remaining)
Mar 25 18:44:14 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:14.104520638Z INFO  nimiq_client         | Consensus: lost - Head: #19315192:MI:fc2eb8a123 - Peers: 21 consensus_established=false block_number=19315192 num_peers=21
Mar 25 18:44:15 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:15.323637532Z INFO  sync                 | Requesting zkp from peer peer_id=12D3KooWKmduaetweNxmzNQyqwYVSSXitwkPyDFHgehNHvJM5TMZ
Mar 25 18:44:24 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:24.104295141Z INFO  nimiq_client         | Consensus: lost - Head: #19315202:MI:a4bde36c7e - Peers: 21 consensus_established=false block_number=19315202 num_peers=21
Mar 25 18:44:34 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:34.104062523Z INFO  nimiq_client         | Consensus: lost - Head: #19315212:MI:d7300f16d0 - Peers: 21 consensus_established=false block_number=19315212 num_peers=21
Mar 25 18:44:36 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:36.297976359Z INFO  sync                 | Requesting zkp from peer peer_id=12D3KooWKmduaetweNxmzNQyqwYVSSXitwkPyDFHgehNHvJM5TMZ
Mar 25 18:44:44 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:44.104243091Z INFO  nimiq_client         | Consensus: lost - Head: #19315222:MI:db99e37c86 - Peers: 21 consensus_established=false block_number=19315222 num_peers=21
Mar 25 18:44:52 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:52.658028885Z INFO  chain_ordering       | New block in Superior chain fork_block_number=19315228 current_block_number=19315230 new_block_number=19315229
Mar 25 18:44:54 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:54.025447744Z INFO  chain_ordering       | New block in Inferior chain fork_block_number=19315228 current_block_number=19315229 new_block_number=19315231
Mar 25 18:44:54 vps-a60b17b1 nimiq-client[2605113]: 2024-03-25T18:44:54.104162914Z INFO  nimiq_client         | Consensus: lost - Head: #19315229:MI:7da6d48b25 - Peers: 21 consensus_established=false block_number=19315229 num_peers=21

There are no errors, the above snippet just repeats itself over and over again. I would expect after reaching a checkpoint block it would snap to consensus state but it keeps syncing without establishing consensus even with 21 peers being connected.

[redmaner]

Update: after running the full node for 24 hours the snippet I shared earlier is still there:

Mar 26 18:46:04 vps-a60b17b1 nimiq-client[2605977]: 2024-03-26T18:46:04.762040668Z INFO  consensus            | Catching up to tip of the chain (now at #19402865, 0 blocks remaining)
Mar 26 18:46:09 vps-a60b17b1 nimiq-client[2605977]: 2024-03-26T18:46:09.537109416Z INFO  sync                 | Requesting zkp from peer peer_id=12D3KooWKmduaetweNxmzNQyqwYVSSXitwkPyDFHgehNHvJM5TMZ
Mar 26 18:46:13 vps-a60b17b1 nimiq-client[2605977]: 2024-03-26T18:46:13.850795793Z INFO  nimiq_client         | Consensus: lost - Head: #19402873:MI:d0d8fb5942 - Peers: 21 consensus_established=false block_number=19402873 num_peers=21
Mar 26 18:46:23 vps-a60b17b1 nimiq-client[2605977]: 2024-03-26T18:46:23.851113548Z INFO  nimiq_client         | Consensus: lost - Head: #19402883:MI:8942109620 - Peers: 21 consensus_established=false block_number=19402883 num_peers=21
Mar 26 18:46:29 vps-a60b17b1 nimiq-client[2605977]: 2024-03-26T18:46:29.709369068Z INFO  sync                 | Requesting zkp from peer peer_id=12D3KooWKmduaetweNxmzNQyqwYVSSXitwkPyDFHgehNHvJM5TMZ
Mar 26 18:46:33 vps-a60b17b1 nimiq-client[2605977]: 2024-03-26T18:46:33.850845669Z INFO  nimiq_client         | Consensus: lost - Head: #19402894:MI:1a2be9b726 - Peers: 21 consensus_established=false block_number=19402894 num_peers=21

It looks like it synced but consensus remains lost. Let me know if I can gather anything from the logs if that helps debug this behavior.

[sisou]