backend/gcs: Service Account Impersonation doesn't work if the original identity is another service account.

[upodroid]

I implemented service account impersonation a while back in #26700 and it works correctly if the original identity is a google user.

However, it doesn't work properly when you are using a Google Service Account as the original identity.

There is a bug present in how cloud.google.com/go/storage(modern/new client) handles ClientOptions supplied to a client but works correctly in google.golang.org/api/storage/v1 (legacy/autogen client)

Terraform Core uses the new client but TPG uses the legacy client.

PoC:

// Modern client in Terraform Core at backend/remote-state/gcs/backend.go

package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"time"

	"cloud.google.com/go/storage"
	"google.golang.org/api/option"
)

func main() {
	ctx := context.Background()

	// Sets your Google Cloud Platform project ID.
	projectID := os.Getenv("GOOGLE_PROJECT")

	// Creates a client.
	client, err := storage.NewClient(ctx, option.ImpersonateCredentials(os.Getenv("GOOGLE_IMPERSONATE_SERVICE_ACCOUNT")))
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}

	// Sets the name for the new bucket.
	bucketName := "foo" // Intentional, expected API response is 409 not 403 from failed impersonation

	// Creates a Bucket instance.
	bucket := client.Bucket(bucketName)

	// Creates the new bucket.
	ctx, cancel := context.WithTimeout(ctx, time.Second*10)
	defer cancel()
	if err := bucket.Create(ctx, projectID, nil); err != nil {
		log.Fatalf("Failed to create bucket: %v", err)
	}

	fmt.Printf("Bucket %v created.\n", bucketName)

}

 REDACTED  MCW0CDP3YY  ~  Desktop  Git  tpg-2252  $  go run main.go 
2021/03/18 15:47:14 Failed to create bucket: Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=mgcp-1192365-landing-zone": impersonate: status code 403: {
  "error": {
    "code": 403,
    "message": "Request had insufficient authentication scopes.",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",
        "domain": "googleapis.com",
        "metadata": {
          "method": "google.iam.credentials.v1.IAMCredentials.GenerateAccessToken",
          "service": "iamcredentials.googleapis.com"
        }
      }
    ]
  }
}
exit status 1

// Legacy/Autogen Client used in TPG
package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"time"

	"google.golang.org/api/option"
	"google.golang.org/api/storage/v1"
)

func main() {
	ctx := context.Background()

	// Sets your Google Cloud Platform project ID.
	projectID := os.Getenv("GOOGLE_PROJECT")

	// Creates a client.
	client, err := storage.NewService(ctx, option.ImpersonateCredentials(os.Getenv("GOOGLE_IMPERSONATE_SERVICE_ACCOUNT")))
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}

	// Creates a Bucket instance.
	sb := &storage.Bucket{
		Name:     "potato", // Expected Response is 409 and not 403 from failed impersonation
		Location: "us",
	}
	// Creates the new bucket.
	ctx, cancel := context.WithTimeout(ctx, time.Second*10)
	defer cancel()

	if _, err := client.Buckets.Insert(projectID, sb).Do(); err != nil {
		log.Fatalf("Failed to create bucket: %v", err)
	}

	fmt.Printf("Bucket %v created.\n", sb.Name)
}

 REDACTED  MCW0CDP3YY  ~  Desktop  Git  tpg-2252  $  go run main.go 
2021/03/18 16:32:11 Failed to create bucket: googleapi: Error 409: Sorry, that name is not available. Please try a different one., conflict
exit status 1

Terraform Logs:

 REDACTED  MCW0CDP3YY  ~  Desktop  Git  REDACTED-Landing-Zone  terraform   initial_build  8✎  3+  $   TF_LOG=debug terraform init
2021/03/18 16:43:11 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
  Use TF_LOG=TRACE to see Terraform's internal logs.
  ----
2021/03/18 16:43:11 [INFO] Terraform version: 0.14.7  
2021/03/18 16:43:11 [INFO] Go runtime version: go1.15.6
2021/03/18 16:43:11 [INFO] CLI args: []string{"/usr/local/Cellar/tfenv/2.0.0/versions/0.14.7/terraform", "init"}
2021/03/18 16:43:11 [DEBUG] Attempting to open CLI config file: /Users/REDACTED/.terraformrc
2021/03/18 16:43:11 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2021/03/18 16:43:11 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2021/03/18 16:43:11 [DEBUG] ignoring non-existing provider search directory /Users/REDACTED/.terraform.d/plugins
2021/03/18 16:43:11 [DEBUG] ignoring non-existing provider search directory /Users/REDACTED/Library/Application Support/io.terraform/plugins
2021/03/18 16:43:11 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2021/03/18 16:43:11 [INFO] CLI command args: []string{"init"}
2021/03/18 16:43:11 [WARN] Log levels other than TRACE are currently unreliable, and are supported only for backward compatibility.
  Use TF_LOG=TRACE to see Terraform's internal logs.
  ----

Initializing the backend...

Error: Failed to get existing workspaces: querying Cloud Storage failed: Get "https://storage.googleapis.com/storage/v1/b/REDACTED-tfstate/o?alt=json&delimiter=%2F&pageToken=&prefix=landing-zone%2F&prettyPrint=false&projection=full&versions=false": impersonate: status code 403: {
  "error": {
    "code": 403,
    "message": "Request had insufficient authentication scopes.",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",
        "domain": "googleapis.com",
        "metadata": {
          "method": "google.iam.credentials.v1.IAMCredentials.GenerateAccessToken",
          "service": "iamcredentials.googleapis.com"
        }
      }
    ]
  }
}

@rileykarson @slevenick How do should we deal with this? Resolve it upstream at https://github.com/googleapis/google-cloud-go or shall I rewrite the gcs backend to use the legacy client?

We also need to bump golang.org/x/oauth2 to a newer version to resolve this hashicorp/terraform-provider-google#8671 in Terraform Core.

[rileykarson]

If there's no bug against googleapis/google-cloud-go I'd suggest filing one, and linking it from here. I've got no big stake in which client library is used to call the GCS API personally, so if someone's working on the feature now it would make sense to switch to the google-go-api-client one. Otherwise, it'll be much easier to take in an upstream fix whenever that happens.

[codyoss]

Just circling back here as well. There is a new impersonate package landing soon being worked on in: googleapis/google-api-go-client#927. This will be first released into a preview tag. Once it lands in a mainline release I would recommend updating to use that package instead relying on the experimental client option. The new package is more flexible with setting scopes which I believe to be the cause of this reported issue.

[upodroid]

@codyoss Thanks for the update, I'll come back and patch this once that new package is released.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.