Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node memory pressure - pod eviction #1254

Open
prom3theu5 opened this issue Jan 20, 2024 · 2 comments
Open

Node memory pressure - pod eviction #1254

prom3theu5 opened this issue Jan 20, 2024 · 2 comments

Comments

@prom3theu5
Copy link

Hi

Currently evaluating some workloads running on a low tier AKS cluster of 3 b2 machines
I've setup robusta minimal without prometheus purely because I need to capture issues with evictions while I evaluate which tier of machine we need for our workloads

Currently a couple of deployments get evicted and rescheduled on another node because of node: low memory
However, these aren't pod oom killed issues, its the node memory threshold I guess which is the reason for the eviction

I expected robusta to alera me when this happened, like it does for pod OOM issues, however it doesn't appear to - i had 50 instances of a pod evicted, and no alerts

Is there not a default playbook setup for this?
do I have to setup a custom playbook, or is what I want not possible with robusta?

thanks

Copy link

Hi 👋, thanks for opening an issue! Please note, it may take some time for us to respond, but we'll get back to you as soon as we can!

  • 💬 Slack Community: Join Robusta team and other contributors on Slack here.
  • 📖 Docs: Find our documentation here.
  • 🎥 YouTube Channel: Watch our videos here.

@aantn
Copy link
Collaborator

aantn commented Feb 24, 2024

Hey, you'll have to setup a custom playbook for this. It's not captured by default due to the potential for a lot of noise on auto-scaled clusters.

I believe there is a Kubernetes event (kubectl get events) that you can use for the custom playbook trigger - can you confirm?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants