Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node spams with error logs when self-terminates because of insufficient storage #4264

Open
alexggh opened this issue Apr 24, 2024 · 2 comments

Comments

@alexggh
Copy link
Contributor

alexggh commented Apr 24, 2024

When node self-terminates because of insufficient with

main storage-monitor: Available space 828MiB for path `/chain-data/chains/versi-networking/db/full` dropped below threshold: 1024MiB , terminating... 

Then it starts spam the logs with around 900K lines of logs, this puts a strain on our logging framework, so we should try to avoid it if possible.

netlink_proto::connection: netlink socket stream shut down
 libp2p_mdns::behaviour: if watch returned an error: rtnetlink socket closed  
 error flushing netlink socket: Custom { kind: Other, error: "A Tokio 1.x context was found, but it is being shutdown.

https://grafana.teleport.parity.io/goto/uWFcFOfSR?orgId=1

Note! This is not a flow that normal validators would exercise, so it is just nice to have fix, if it can be done with low effort.

@BulatSaif
Copy link
Contributor

BulatSaif commented Apr 24, 2024

A Tokio 1.x context was found

This error not always caused by insufficient space. Usually, the node just exits with an error if you have insufficient space.

I've encountered it several times by simply killing the node with "Ctrl+C" on startup (enough disk space). This error blocks the terminal, requiring you to log in with a second terminal to kill the process.

@sandreim
Copy link
Contributor

What we need is graceful node shutdown for such situations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants