Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] Running HEAD causes issue trying to authenticate with SCP when not using SCP #932

Open
lod opened this issue May 8, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@lod
Copy link

lod commented May 8, 2024

Describe the bug
Attempting to copy files from the local filesystem to a Google storage bucket.

To Reproduce

  1. skyplane init - GCP setup, no other systems.
  2. skyplane cp -r /local/directory "gs://remote-bucket/"
  3. Error

Expected behavior
Files copied to GCP - I don't care about other cloud systems.

Screenshots

/  ___| | / /\ \ / / ___ \ |     / _ \ | \ | ||  ___|
\ `--.| |/ /  \ V /| |_/ / |    / /_\ \|  \| || |__
 `--. \    \   \ / |  __/| |    |  _  || . ` ||  __|
/\__/ / |\  \  | | | |   | |____| | | || |\  || |___
\____/\_| \_/  \_/ \_|   \_____/\_| |_/\_| \_/\____/

Azure resource group and UMI name not configured correctly. Please reinit Azure with `skyplane init --reinit-azure`.
Logging to: /tmp/skyplane/transfer_logs/20240508_072013-4b456028/client.log
Uncaught exception: (AssertionError) SCP Credentials not found
Please check the log file for more information, and ensure to include it if reporting an issue on Github.

Transfer client log
(parent paths trimmed, shouldn't be relevant)

07:20:13 [ERROR] Uncaught exception: AssertionError: SCP Credentials not found
07:20:13 [ERROR] Traceback:
Traceback (most recent call last):
  File ".../skyplane/env/bin/skyplane", line 8, in <module>
    sys.exit(app())
  File ".../skyplane/env/lib/python3.10/site-packages/typer/main.py", line 326, in __call__
    raise e
  File ".../skyplane/env/lib/python3.10/site-packages/typer/main.py", line 309, in __call__
    return get_command(self)(*args, **kwargs)
  File ".../skyplane/env/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File ".../skyplane/env/lib/python3.10/site-packages/typer/core.py", line 723, in main
    return _main(
  File ".../skyplane/env/lib/python3.10/site-packages/typer/core.py", line 193, in _main
    rv = self.invoke(ctx)
  File ".../skyplane/env/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File ".../skyplane/env/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File ".../skyplane/env/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File ".../skyplane/env/lib/python3.10/site-packages/typer/main.py", line 692, in wrapper
    return callback(**use_params)
  File ".../skyplane/skyplane/cli/cli_transfer.py", line 471, in cp
    return run_transfer(src, dst, recursive, debug, multipart, confirm, max_instances, max_connections, solver, "cp")
  File ".../skyplane/skyplane/cli/cli_transfer.py", line 351, in run_transfer
    cli = SkyplaneCLI(src_region_tag=src_region_tag, dst_region_tag=dst_region_tag, args=args)
  File ".../skyplane/skyplane/cli/cli_transfer.py", line 67, in __init__
    self.client = skyplane.SkyplaneClient(
  File ".../skyplane/skyplane/api/client.py", line 65, in __init__
    self.provisioner = Provisioner(
  File ".../skyplane/skyplane/api/provisioner.py", line 77, in __init__
    self._make_cloud_providers()
  File ".../skyplane/skyplane/api/provisioner.py", line 92, in _make_cloud_providers
    self.scp = compute.SCPCloudProvider(auth=self.scp_auth)
  File ".../skyplane/skyplane/compute/scp/scp_cloud_provider.py", line 54, in __init__
    self.network = network if network else SCPNetwork(self.auth)
  File ".../skyplane/skyplane/compute/scp/scp_network.py", line 38, in __init__
    self.scp_client = SCPClient()
  File ".../skyplane/skyplane/compute/scp/scp_utils.py", line 117, in __init__
    assert os.path.exists(self.credentials), "SCP Credentials not found"
AssertionError: SCP Credentials not found

Environment info (please complete the following information):

  • OS: Ubuntu 22.04.4 LTS (Jammy Jellyfish)
  • Python version 3.10.12
  • Skyplane version: git head - 4602d9c

Additional context

When that problem was fixed (disabled) I ran into a problem with the scp load quota.
skyplane/planner/planner.py:53 Attempts to load and json parse the scp_quota file. My file (default init) exists but is empty, the json parse fails.

Workaround

The following changes allowed me to proceed by disabling SCP, they are a reasonable work around but they don't actually attempt to fix the problem properly.

diff --git a/skyplane/api/provisioner.py b/skyplane/api/provisioner.py
index 7e09817..d3a4048 100644
--- a/skyplane/api/provisioner.py
+++ b/skyplane/api/provisioner.py
@@ -89,7 +89,7 @@ class Provisioner:
         self.azure = compute.AzureCloudProvider(auth=self.azure_auth)
         self.gcp = compute.GCPCloudProvider(auth=self.gcp_auth)
         self.ibmcloud = compute.IBMCloudProvider(auth=self.ibmcloud_auth)
-        self.scp = compute.SCPCloudProvider(auth=self.scp_auth)
+        self.scp = None

     def init_global(self, aws: bool = True, azure: bool = True, gcp: bool = True, ibmcloud: bool = True, scp: bool = True):
         """
diff --git a/skyplane/planner/planner.py b/skyplane/planner/planner.py
index dde2846..2e75ace 100644
--- a/skyplane/planner/planner.py
+++ b/skyplane/planner/planner.py
@@ -48,9 +48,6 @@ class Planner:
             if os.path.exists(gcp_quota_path):
                 with gcp_quota_path.open("r") as f:
                     quota_limits["gcp"] = json.load(f)
-            if os.path.exists(scp_quota_path):
-                with scp_quota_path.open("r") as f:
-                    quota_limits["scp"] = json.load(f)
         self.quota_limits = quota_limits

         # Loading the vcpu information - a dictionary of dictionaries
@lod lod added the bug Something isn't working label May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant