New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
out of memory #395
Comments
It clearly looks like a nasty memory leak. Just to try to pinpoint what's going on. You're using RRD provider right, and you just let the service run with a refresh interval (if yes what was the value)? |
Hi Vincent, yes I am using the RRD provider. The refresh interval for the UI is set to 10s, this is default I guess. The memory leak occurs purely on startup of the server, even if it is running completely without any client interaction. Can I provide you with more information to help pinning down the cause for the memory leak? Best, |
hey @akohlbecker, Quick additional questions:
Regards, |
The refresh interval for the rrd provider was set to I will report long term results from this settings change tomorrow. |
This setting unit is seconds: https://docs.facette.io/latest/api/providers/#create-a-provider It would be surprising that raising this interval fixes the issue. It might take longer to trigger though. |
Hi Vincent, your expectation was correct, the memory consumption has increased over night and is now at about 10 GB. I checked the rrd folders for symlinks and found none. The debug log contains many entries like these (ellypsed here):
apart from these 588313 lines after running facette for 10 minutes with an rrd provider refresh interval of 500, the log only has these entries:
|
Which version are you running, the latest release or a build from What's your platform/architecture, linux/amd64? I built a custom version yesterday having a pprof HTTP endpoint that might would allow us to visualize heap usage while running the service. I'll try to push it in a dedicated branch tonight but I can build the binary for you to test if you want. |
I am running version 0.5.1 on linux/amd64 (4.9.0-15-amd64 #1 SMP Debian 4.9.258-1 (2021-03-08) x86_64 GNU/Linux) It would be great if you could build the binary for me. TNX |
Hi @akohlbecker, Sorry for the delay here. I just pushed changes to a dedicated branch that registers debugging pprof endpoints to the web server, see 593ce3f. Here comes a .deb file embedding those changes (note: had to gzip it to make GitHub accept it 🤷 ): Once installed and the issue triggered, you should be able to visualize heap information from the running service using:
If you could extract it for me, it would be great too:
|
Hi Vincent, thank you for the binary. BTW: Since I've set the refresh interval for the rrd provider to 500 I no longer have problems. In installed the debug build anyway and here is the pprof output: Cheers |
After starting the service facette eats up ~ 19GB RAM and terminates with
fatal error: runtime: out of memory
Please find thew corresponding systemlog attached to this ticket.
facette-outofmem.log
The text was updated successfully, but these errors were encountered: