Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SKY-272] [bug] Error with service account key deletion for GCP #922

Open
sarahwooders opened this issue Oct 16, 2023 · 0 comments
Open
Labels
bug Something isn't working

Comments

@sarahwooders
Copy link
Contributor

sarahwooders commented Oct 16, 2023

Describe the bug
Service account key created by skyplane need to be deleted (since no more than 10 keys are allowed by GCP), however this causes an error during init.

To Reproduce
Run skyplane init --reinit-gcp.

Transfer client log
In the log output from Skyplane, please upload the debug log from the CLI. You can find the path to the file in the log output:

╰─ skyplane init --reinit-gcp                                                                                                   ─╯
 _____ _   ____   _______ _       ___   _   _  _____
/  ___| | / /\ \ / / ___ \ |     / _ \ | \ | ||  ___|
\ `--.| |/ /  \ V /| |_/ / |    / /_\ \|  \| || |__
 `--. \    \   \ / |  __/| |    |  _  || . ` ||  __|
/\__/ / |\  \  | | | |   | |____| | | || |\  || |___
\____/\_| \_/  \_/ \_|   \_____/\_| |_/\_| \_/\____/

13:23:36 [DEBUG] Found existing configuration file at /Users/sarahwooders/.skyplane/config, loading

Configuring GCP:
    Do you want to configure GCP support in Skyplane? [Y/n]: Y
    GCP credentials will be re-initialized
/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/google/auth/_default.py:76: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. See the following page for troubleshooting: https://cloud.google.com/docs/authentication/adc-troubleshooting/user-creds.
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
    GCP credentials found in GCP CLI
    Do you want to enable GCP support in Skyplane? [Y/n]: Y
    Enter the GCP project ID [skyplane-broadcast]:
    Using GCP service account skyplane-manual
13:23:45 [WARN]  Service account skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com has too many keys. Deleting stale keys
to create new key.
    Error saving GCP region config
    Failed to delete key projects/skyplane-broadcast/serviceAccounts/skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com/keys/82ae2078a0798c1d83195bc37fbd2cbc29c6c179: <HttpError 400 when requesting https://iam.googleapis.com/v1/projects/skyplane-broadcast/serviceAccounts/skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com/keys/82ae2078a0798c1d83195bc37fbd2cbc29c6c179?alt=json returned "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.". Details: "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.">
Traceback (most recent call last):
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/compute/gcp/gcp_auth.py", line 170, in get_service_account_key
    service.projects().serviceAccounts().keys().delete(name=key["name"]).execute()
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://iam.googleapis.com/v1/projects/skyplane-broadcast/serviceAccounts/skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com/keys/82ae2078a0798c1d83195bc37fbd2cbc29c6c179?alt=json returned "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.". Details: "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/cli/cli_init.py", line 365, in load_gcp_config
    auth.save_region_config()
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/utils/imports.py", line 33, in wrapped
    return fn(*modules_imported, *args, **kwargs)
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/compute/gcp/gcp_auth.py", line 36, in save_region_config
    service_account_credentials_file = self.service_account_credentials  # force creation of file
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/compute/gcp/gcp_auth.py", line 93, in service_account_credentials
    self._service_credentials_file = self.get_service_account_key(self._service_account_email)
  File "/Users/sarahwooders/repos/sky-atc/env/lib/python3.10/site-packages/skyplane/compute/gcp/gcp_auth.py", line 173, in get_service_account_key
    raise ValueError(f"Failed to delete key {key['name']}: {e}")
ValueError: Failed to delete key projects/skyplane-broadcast/serviceAccounts/skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com/keys/82ae2078a0798c1d83195bc37fbd2cbc29c6c179: <HttpError 400 when requesting https://iam.googleapis.com/v1/projects/skyplane-broadcast/serviceAccounts/skyplane-manual@skyplane-broadcast.iam.gserviceaccount.com/keys/82ae2078a0798c1d83195bc37fbd2cbc29c6c179?alt=json returned "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.". Details: "Service Account Key 82ae2078a0798c1d83195bc37fbd2cbc29c6c179 is a system managed key and can not be deleted.">

    Disabling Google Cloud support

Environment info (please complete the following information):

  • OS: Mac
  • Skyplane version: 0.3.2

SKY-272

@sarahwooders sarahwooders added the bug Something isn't working label Oct 16, 2023
@sarahwooders sarahwooders changed the title [bug] Error with service account key deletion for GCP [SKY-272] [bug] Error with service account key deletion for GCP Oct 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant