Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

promdump: improve handling for 503 Service Unavailable issues #98

Open
ionthegeek opened this issue Nov 15, 2023 · 1 comment
Open

promdump: improve handling for 503 Service Unavailable issues #98

ionthegeek opened this issue Nov 15, 2023 · 1 comment
Labels
enhancement New feature or request promdump Issues related to the prometheus-remote-backfill tools

Comments

@ionthegeek
Copy link
Contributor

Prometheus will return 503 Service Unavailable if a PromQL query times out or aborts. This is particularly prevalent for environments with very large numbers of nodes, where the tserver_export and node_export queries may return a lot of data.

The promdump utility should be enhanced to improve handling for this condition. For example, by adding retry logic and / or reducing the batch size to avoid overloading the Prometheus server.
@ionthegeek ionthegeek added enhancement New feature or request promdump Issues related to the prometheus-remote-backfill tools labels Nov 15, 2023
@eugeneckim
Copy link
Contributor

Recent thread which touched on this.
https://yugabyte.slack.com/archives/C072YT48W57/p1715611566420589?thread_ts=1715500866.247639&cid=C072YT48W57

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request promdump Issues related to the prometheus-remote-backfill tools
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ionthegeek @eugeneckim