Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report the collector as unhealthy if the number of running tasks is high #8507

Open
3 tasks
fniessink opened this issue Apr 16, 2024 · 0 comments
Open
3 tasks
Labels
Tech Technical change without significant user impact
Projects

Comments

@fniessink
Copy link
Member

fniessink commented Apr 16, 2024

Add an environment variable COLLECTOR_RUNNING_TASKS_HEALTHY_THRESHOLD and report the container as unhealthy if the number of running tasks (as tracked by Collector.running_tasks) exceeds the threshold.

Tasks:

  • Add the environment variable to base_collectors.config
  • Default value: 120. Rationale using the default config values: the collector starts max 30 tasks every 20 seconds that timeout after max 120 seconds so the theoretical maximum number of running tasks is (120/20) x 30 = 180. Use 2/3 of that. Add the rationale to the code and docs.
  • Change the Collector.record_health method to not update the timestamp when the number of running tasks exceeds the threshold.
@fniessink fniessink added the Tech Technical change without significant user impact label Apr 16, 2024
@fniessink fniessink added this to Backlog in Backlog via automation Apr 16, 2024
@fniessink fniessink moved this from Backlog to Ready in Backlog Apr 16, 2024
@fniessink fniessink changed the title Report the collector container as unhealthy if the number of running tasks is high Report the collector as unhealthy if the number of running tasks is high Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Tech Technical change without significant user impact
Projects
Backlog
Ready
Development

No branches or pull requests

1 participant