Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too light log severities for a serious problems #1223

Closed
sammy007 opened this issue Apr 18, 2024 · 2 comments
Closed

Too light log severities for a serious problems #1223

sammy007 opened this issue Apr 18, 2024 · 2 comments

Comments

@sammy007
Copy link
Contributor

I have borrowed that log from another issue for demonstration:

WARN [11-30|11:10:27.352] Synchronisation failed, dropping peer    peer=1673b28af0061121f1ec941aae813b2c5907d3b7fcde0746aee66d28ce2e8247 err="retrieved hash chain is invalid: invalid merkle root (remote: 55f9a3e801a05d3330f06aa1a9ba9a09fe7979230d50846bb9d886c96383755e local: e18a8fbe060ebefdc86cd2f2a5d0c14f70f6dfcc7252efe232c4c867ef37a01f) dberr: %!w(<nil>)"
WARN [11-30|11:10:27.352] Synchronisation failed, retrying         peer=1673b28af0061121f1ec941aae813b2c5907d3b7fcde0746aee66d28ce2e8247 err="peer is unknown or unhealthy"
INFO [11-30|11:10:34.342] Got new milestone from heimdall          start=50,570,307 end=50,570,319 hash=0xf1d3d69bd5d203c84d9e697a47ef078b91bea1b230508f3e6013aefde6a6aa1f
WARN [11-30|11:10:34.343] unable to handle whitelist milestone     err="missing blocks"
INFO [11-30|11:10:46.343] Got new milestone from heimdall          start=50,570,307 end=50,570,319 hash=0xf1d3d69bd5d203c84d9e697a47ef078b91bea1b230508f3e6013aefde6a6aa1f
WARN [11-30|11:10:46.343] unable to handle whitelist milestone     err="missing blocks"
INFO [11-30|11:10:58.343] Got new milestone from heimdall          start=50,570,307 end=50,570,319 hash=0xf1d3d69bd5d203c84d9e697a47ef078b91bea1b230508f3e6013aefde6a6aa1f
WARN [11-30|11:10:58.343] unable to handle whitelist milestone     err="missing blocks"

Basically it's a catastrophic case, it indicates that node is in serious trouble, it must be an ERROR or even CRITICAL event, instead Bor is logging it as just WARNING. Please consider changing severity in these cases.

Heimdall does the opposite which is also an issue maticnetwork/heimdall#1159

@Raneet10
Copy link
Member

Hi! There are two types of WARN logs here. I'm not sure which one you're concerned about so I'll cover both

WARN [11-30|11:10:27.352] Synchronisation failed, dropping peer    peer=1673b28af0061121f1ec941aae813b2c5907d3b7fcde0746aee66d28ce2e8247 err="retrieved hash chain is invalid: invalid merkle root (remote: 55f9a3e801a05d3330f06aa1a9ba9a09fe7979230d50846bb9d886c96383755e local: e18a8fbe060ebefdc86cd2f2a5d0c14f70f6dfcc7252efe232c4c867ef37a01f) dberr: %!w(<nil>)"
WARN [11-30|11:10:27.352] Synchronisation failed, retrying         peer=1673b28af0061121f1ec941aae813b2c5907d3b7fcde0746aee66d28ce2e8247 err="peer is unknown or unhealthy"

This doesn't necessarily indicate something's wrong with the node, it could be possible it got connected to a "bad" peer. In this case you can just remove this peer or reduce maxpeers count in case it repeatedly connects with it. If you see this error popping up while syncing with other peers, then there's something wrong and I'll suggest in this case you rewind your node a couple of blocks and resync.
Since it is not trivial to distinguish which is the case here, tagging these logs under WARN seems good enough.

WARN [11-30|11:10:34.343] unable to handle whitelist milestone err="missing blocks"

This just indicates that your node isn't able to whitelist milestone (i.e. finalized) blocks since it isn't in sync and doesn't have the blocks in its local DB.

Hope this helps!

@sammy007
Copy link
Contributor Author

Okay, seems reasonable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants