Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crawler pod memory padding + auto scaling #1631

Merged
merged 4 commits into from Mar 28, 2024
Merged

Crawler pod memory padding + auto scaling #1631

merged 4 commits into from Mar 28, 2024

Conversation

ikreymer
Copy link
Member

@ikreymer ikreymer commented Mar 27, 2024 •
edited

  • set memory limit to 1.2x memory request to provide extra padding and avoid OOM
  • attempt to resize crawler pods by 1.2 when exceeding 90% of available memory
  • do a 'soft OOM' (send extra SIGTERM) to pod when reaching 100% of requested memory, resulting in faster graceful restart, but avoiding a system-instant OOM Kill
  • Fixes [Feature]: Allow crawler pods to be scaled up if they need more memory #1632

@ikreymer ikreymer requested a review from tw4l March 27, 2024 15:48
@ikreymer
memory limit padding + auto-scale crawler pods:
14d637b 
- set memory limit to 1.2x memory request to provide extra padding and avoid OOM
- attempt to resize crawler pods by 1.2 when exceeding 90% of available memory
- do a 'soft OOM' (send extra SIGTERM) to pod when reaching 100% of requested memory, resulting in faster graceful restart
tw4l
tw4l approved these changes Mar 27, 2024
View reviewed changes
Copy link
Contributor

@tw4l tw4l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally and works well! Both the resizing and OOM handling are big improvements.

I also pushed a commit to remove some of the page data debug logging leftover from development that was cluttering the operator logs.

@ikreymer
add check for has_pod_metrics
047c96a
@ikreymer
signal sending fixes:
ba46d81 
- add exec permission to send signal
- add signalAtMem to avoid resending SIGTERM multiple times
@ikreymer ikreymer merged commit 3438133 into main Mar 28, 2024
4 checks passed
@ikreymer ikreymer deleted the auto-scale-crawler-pods branch March 28, 2024 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tw4l tw4l tw4l approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature]: Allow crawler pods to be scaled up if they need more memory
2 participants
@ikreymer @tw4l