Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pyroscope retention period config #109

Open
rmn-lux opened this issue Nov 15, 2023 · 2 comments
Open

Pyroscope retention period config #109

rmn-lux opened this issue Nov 15, 2023 · 2 comments

Comments

@rmn-lux
Copy link

rmn-lux commented Nov 15, 2023

Hello, thank you for the coroot. I found out about this great product in an article on Habr.

I have a default installation in kubernetes via helm chart. Faced a problem that in pyroscope either retention doesn't work or I don't understand something. The space is clogged with metrics very quickly and is not cleared.

The configuration was default, I reduced retention but eventually all the allocated space is clogged and everything stands at a standstill.

image

On the screenshot above is an example: in the morning at 9.00 a clean install was done, and by the evening at 18.00 it was already 3 GB. Can you please tell me how to configure retention correctly so that it works and the metrics take up less space? Thanks.

P.S. I realize that the problem is not in coroot itself, but in a third-party product, but I hope you can tell me how to properly work with metrics in pyroscope or suggest some documentation on this issue.

UPD. for 21 hours of work accumulated logs of 8 GB already. there are no production services, test microservices are installed from https://github.com/GoogleCloudPlatform/microservices-demo.

@apetruhin
Copy link
Member

@rmn-lux, thank you for the report.
It appears that Pyroscope does not remove old data (grafana/pyroscope#1977).
We are currently exploring ways to fix or implement a workaround for this issue.

@def
Copy link
Member

def commented Jan 23, 2024

@rmn-lux, starting from version 0.27, Coroot uses ClickHouse to store profiling data. The default TTL is set to 7 days, but you can adjust it if necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants