Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

notebooks should not have ephemeral storage limit #246

Open
samos123 opened this issue Oct 7, 2023 · 1 comment
Open

notebooks should not have ephemeral storage limit #246

samos123 opened this issue Oct 7, 2023 · 1 comment
Assignees
@samos123

Comments

@samos123
Copy link
Contributor

samos123 commented Oct 7, 2023 

  Warning  Evicted              60s                kubelet            Pod ephemeral local storage usage exceeds the total limit of containers 100Gi.

This can be frustating because in many cases the node had more than 100Gi ephemeral storage available and I lose all my notebook progress when this happens
@samos123 samos123 self-assigned this Oct 7, 2023
@samos123
Copy link
Contributor Author

This turned out to be a K8s bug that if one of the sidecars sets a limit then all containers inherit that limit

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@samos123 samos123

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@samos123