limit the 'beacon chain is in inactivity leak' message

[rolfyone]

		2024-03-12 08:19:00.047	
{"@timestamp":"2024-03-11T22:19:00,014","level":"INFO","thread":"TimeTick-1","class":"teku-event-log","message":"Syncing     *** Target slot: 7807295, Head slot: 7772295, Remaining slots: 35000, Connected peers: 95","throwable":""}

2024-03-12 08:18:59.796	
{"@timestamp":"2024-03-11T22:18:59,732","level":"INFO","thread":"beaconchain-async-7","class":"AbstractEpochProcessor","message":"Beacon chain is in inactivity leak","throwable":""}
2024-03-12 08:18:58.543	
{"@timestamp":"2024-03-11T22:18:58,323","level":"INFO","thread":"beaconchain-async-7","class":"AbstractEpochProcessor","message":"Beacon chain is in inactivity leak","throwable":""}
2024-03-12 08:18:56.788	
{"@timestamp":"2024-03-11T22:18:56,640","level":"INFO","thread":"beaconchain-async-7","class":"AbstractEpochProcessor","message":"Beacon chain is in inactivity leak","throwable":""}
2024-03-12 08:18:55.536	
{"@timestamp":"2024-03-11T22:18:55,303","level":"INFO","thread":"beaconchain-async-7","class":"AbstractEpochProcessor","message":"Beacon chain is in inactivity leak","throwable":""}
2024-03-12 08:18:53.531	
{"@timestamp":"2024-03-11T22:18:53,327","level":"INFO","thread":"beaconchain-async-7","class":"AbstractEpochProcessor","message":"Beacon chain is in inactivity leak","throwable":""}
2024-03-12 08:18:52.027	
{"@timestamp":"2024-03-11T22:18:51,899","level":"INFO","thread":"beaconchain-async-7","class":"AbstractEpochProcessor","message":"Beacon chain is in inactivity leak","throwable":""}
2024-03-12 08:18:50.524	
{"@timestamp":"2024-03-11T22:18:50,288","level":"INFO","thread":"beaconchain-async-7","class":"AbstractEpochProcessor","message":"Beacon chain is in inactivity leak","throwable":""}
2024-03-12 08:18:49.021	
{"@timestamp":"2024-03-11T22:18:48,774","level":"INFO","thread":"beaconchain-async-7","class":"AbstractEpochProcessor","message":"Beacon chain is in inactivity leak","throwable":""}
2024-03-12 08:18:48.019	
{"@timestamp":"2024-03-11T22:18:48,014","level":"INFO","thread":"TimeTick-1","class":"teku-event-log","message":"Syncing     *** Target slot: 7807294, Head slot: 7772295, Remaining slots: 34999, Connected peers: 95","throwable":""}

It doesnt seem totally consistnet, but we only need this every epoch once...

[mehdi-aouadi]

Isn't this a duplicate of #7329 fixed by #7788?

[mehdi-aouadi]

@rolfyone I'm unable to reproduce this behaviour. In which box did you find these logs?
I checked our goerli nodes and the logs are well throttled as expected (every epoch).
The two boxes where I found the same logs are goerli-vc-03 and goerli-vc-01. In both the logs happen every epoch only.
Looked at the code too, I didn't see any initialisation issue. Curious to know when did that happen...

[lucassaldanha]

It is possible that this was fixed by #7788. If we can't find it happening on any instance after we released #7788 we can probably close this issue.

[rolfyone]

IIRC theres an initialization issue where if the epoch (slot?) was unset then it'd continue to spam, we may just need to cater for that case where it's unset...

[zilm13]

I saw it when testing PeerDAS, so is not fixed probably with #7788