1.26 on hash consumed almost all memory and CPU, did not process blocks

[yorickdowne]

Description

Nethermind 1.26 did not process blocks and gradually consumed all memory, and had high CPU.

A restart fixed it, it caught up again. It would not quit cleanly however, Docker reaped it after the 5min configured timeout for container stop.

CL is Nimbus v24.4.0

Logs don't appear to show a clear root cause.

Nethermind is started with these pruning parameters. It was not running a Full Prune at the time. --Pruning.FullPruningMaxDegreeOfParallelism=3 --Pruning.FullPruningTrigger=VolumeFreeSpace --Pruning.FullPruningThresholdMb=375810 --Pruning.CacheMb=4096 --Pruning.FullPruningMemoryBudgetMb=16384 --Init.StateDbKeyScheme=HalfPath

Full startup parameters as shown by ps auxww:

/nethermind/nethermind --datadir /var/lib/nethermind --Init.WebSocketsEnabled true --Network.DiscoveryPort 30303 --Network.P2PPort 30303 --Network.MaxActivePeers 50 --HealthChecks.Enabled true --HealthChecks.UIEnabled true --JsonRpc.Enabled true --JsonRpc.Host 0.0.0.0 --JsonRpc.Port 8545 --JsonRpc.WebSocketsPort 8546 --JsonRpc.EngineHost 0.0.0.0 --JsonRpc.EnginePort 8551 --JsonRpc.AdditionalRpcUrls=http://127.0.0.1:1337|http|admin --JsonRpc.JwtSecretFile=/var/lib/nethermind/ee-secret/jwtsecret --Metrics.Enabled true --Metrics.PushGatewayUrl  --Metrics.ExposeHost 0.0.0.0 --Metrics.ExposePort 6060 --Pruning.FullPruningCompletionBehavior AlwaysShutdown --log info --config mainnet --JsonRpc.EnabledModules Web3,Eth,Subscribe,Net,Health,Parity,Proof,Trace,TxPool --Pruning.FullPruningMaxDegreeOfParallelism=3 --Pruning.FullPruningTrigger=VolumeFreeSpace --Pruning.FullPruningThresholdMb=375810 --Pruning.CacheMb=4096 --Pruning.FullPruningMemoryBudgetMb=16384 --Init.StateDbKeyScheme=HalfPath

Steps to Reproduce

Unsure

Desktop (please complete the following information):
Please provide the following information regarding your setup:

• Operating System: Linux
• Version: 1.26.0+0068729c
• Installation Method: Docker
• Consensus Client: Nimbus v24.4.0

Logs

Nimbus logs

Logs of Nethermind in its failure state

Logs of Nethermind after restart, when it catches up

nimbus.log.gz

nm-hang.log.gz

nm-catchup.log.gz

[yorickdowne]

This happened again on another machine with a Lodestar CL. It's unlikely be an interop issue then. I will remove --Pruning.CacheMb=4096 to see whether that makes a difference.

[asdacap]

Hi, do you have log with the Lodestar CL? Also, can you double check that the CL is synced?

[yorickdowne]

We have the logs in Loki, let me see what I can pull out. The CL is synced, yes.

Since removing --Pruning.CacheMb=4096 from all our nodes we haven't seen a failure in 2 days, where previously we had 2 different ones fail in 2 days.

[asdacap]

May I know why --Init.StateDbKeyScheme=HalfPath is on when running hash? Do you plan to migrate via full pruning?

[yorickdowne]

Yes and yes. The intent was to allow the auto prune to migrate.

Have not seen another failure since removing the pruning cache parameter

[asdacap]

Did not reproduce when forward syncing. Problem may come from outside block processing.

[yorickdowne]

I’ve seen this once more, it just took those 2 weeks.

I’ve configured one of my servers with the cache parameter and debug logs, to hopefully catch this when it happens

[asdacap]

Are those running hash also or halfpath?

[yorickdowne]

This is hash, with the HalfPath parameter to convert during pruning.

[asdacap]

Are these node validator?

[yorickdowne]

Yes, they are part of my Lido setup. Nimbus or Lodestar depending on the server; 1,000 validators via Vouch per cluster. Means there is a decent amount of block building going on.

[asdacap]

Hmm... simply re-running blocks with block producer's block processor does not work. Need actual block production.