[Feature]: Allow crawler pods to be scaled up if they need more memory #1632
Assignees
Labels
enhancement
New feature or request
Comments
ikreymer
added a commit
that referenced
this issue
Mar 28, 2024
- set memory limit to 1.2x memory request to provide extra padding and avoid OOM - attempt to resize crawler pods by 1.2x when exceeding 90% of available memory - do a 'soft OOM' (send extra SIGTERM) to pod when reaching 100% of requested memory, resulting in faster graceful restart, but avoiding a system-instant OOM Kill - Fixes #1632 --------- Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What change would you like to see?
Currently, crawler pods have a fixed resource usage. With our custom operator and metrics api, we can dynamically give crawler pods more memory, and/or restart them before they are killed by the system.
Context
Crawling some sites will inevitable require more memory than others. With ability to resize the pods (with a restart for now, eventually without), crawling can ideally be more smooth and avoid data corruption due to restarts.
The idea is to gracefully stop the crawler pods and increase memory if >90% of memory is used.
We can also set the memory limit > memory request, and less gracefully restart the pod if >100% requested memory is used (soft OOM), but before the pod is killed by OOM Killer
The text was updated successfully, but these errors were encountered: