irregular application crash

[fernando-galaxystaking]

Our application (v 3.3.3) has now crashed several times on different hardware with the following error message you see in attach.

Due to the automatic restart of the service, the application comes back, but in most cases it crashes again immediately or reports a db-error.
I then only get the node to run by importing a statesync db and sync the delta.

What means the error message? Where does this error message come from? Disk, RAM, CPU, Connection?
error.log

[adu-crypto]

it seems the type assertion here has failed and trigger a panic:
https://github.com/cosmos/cosmos-sdk/blob/master/server/logger.go#L51

[alexanderbez]

Yeah something isn't adding up here. The stacktrace seems to indicate something else. It does seem like some sort of memory corruption or just a memory related issue. Does this happen consistently? Or did you only see this issue once?

[fernando-galaxystaking]

Litte Background:
We have several (>10) nodes / servers in use to map our infrastructure, since we have a total of 3 validators, 6 sentrys, Web-Server (Homepage), 2 backup servers with running chain, statesync server to make snapshots (https://statesync.galaxystaking.space) and watchdog for failure.

This problem has now occurred about 4-5 times on different servers.
However, as a precaution, they were completely replaced by the hoster, but problem still exists sporadically.

In the next Step i would let replace the RAM with ECC-RAM. This would solve the problem right?

[alexanderbez]

Is it deterministic? Like does it always happen at a specific block height? Or is it somewhat random?

[fernando-galaxystaking]

Unfortunately not predictable. It's always on a different block.
In the event of a failure, our watchdog kicked in, the validator is pushed to a backup server and continues to run there.
The "defective" one is then loaded with a fresh statesync DB (to save space and rule out possible DB errors) and is available again as a backup.
It is quite possible that without a DB error, the chain will start up again after a few minutes and continue running. But I won't let it get that far, because the number of missed blocks is important.
Next time (if comes :D) i will use / check the old DB after moved to backup server.

[fernando-galaxystaking]

I'm currently still wondering whether ECC would solve this problem?