Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multizone clusters with GPU_NODE_MIN_SIZE=1 must have a GPU quota of at least 2. #367

Open
willgraf opened this issue Jun 29, 2020 · 1 comment
Labels
wontfix This will not be worked on

Comments

@willgraf
Copy link
Contributor

willgraf commented Jun 29, 2020

Describe the bug
Deploying a cluster in multiple zones requires a GPU quota equal to the number of zones if GPU_NODE_MIN_SIZE is at least 1. If the GPU quota is only 1, a multizone cluster will fail with the error:

ERROR: (gcloud.container.node-pools.create) ResponseError: code=403, message=Insufficient regional quota to satisfy request: resource "PREEMPTIBLE_NVIDIA_V100_GPUS": request requires '2.0' and is short '1.0'. project has a quota of '1.0' with '1.0' available. View and manage quotas at https://console.cloud.google.com/iam-admin/quotas?usage=USED.

To Reproduce
Steps to reproduce the behavior:

  1. During cluster configuration, select a GPU type with a quota of 1.
  2. Select a minimum GPU node count of at least 1.
  3. Select a multizone cluster configuration.
  4. Create the cluster.
  5. See the error in the failed cluster creation output.

Expected behavior
Ideally, the cluster will not allow the user to start creating a cluster with a bad configuration.

@willgraf willgraf added the bug Something isn't working label Jun 29, 2020
@willgraf willgraf changed the title Multizone clusters must have a GPU quota of at least 2. Multizone clusters with GPU_NODE_MIN_SIZE=1 must have a GPU quota of at least 2. Jul 1, 2020
@willgraf
Copy link
Contributor Author

willgraf commented Oct 5, 2020

The solution to this bug is tracked in #335, though I will leave this issue open as a documented warning until it is resolved.

@willgraf willgraf added the wontfix This will not be worked on label Oct 5, 2020
@willgraf willgraf removed the bug Something isn't working label Dec 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

1 participant