Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crawler pod memory padding + auto scaling #1631

Merged
merged 4 commits into from Mar 28, 2024
Merged

Crawler pod memory padding + auto scaling #1631

merged 4 commits into from Mar 28, 2024

Commits on Mar 27, 2024

  1. memory limit padding + auto-scale crawler pods: 

    - set memory limit to 1.2x memory request to provide extra padding and avoid OOM
- attempt to resize crawler pods by 1.2 when exceeding 90% of available memory
- do a 'soft OOM' (send extra SIGTERM) to pod when reaching 100% of requested memory, resulting in faster graceful restart
    @ikreymer
    ikreymer committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    14d637b View commit details
    Browse the repository at this point in the history

  2. Remove page debug logging cluttering logs

    @tw4l
    tw4l committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    378d58a View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. add check for has_pod_metrics

    @ikreymer
    ikreymer committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    047c96a View commit details
    Browse the repository at this point in the history

  2. signal sending fixes: 

    - add exec permission to send signal
- add signalAtMem to avoid resending SIGTERM multiple times
    @ikreymer
    ikreymer committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    ba46d81 View commit details
    Browse the repository at this point in the history