storage: Service Account Impersonation doesn't work if the original identity is another service account.

[upodroid]

Crossposting from hashicorp/terraform#28139

There is a bug present in how cloud.google.com/go/storage(modern/new client) handles ClientOptions supplied to a client but works correctly in google.golang.org/api/storage/v1 (legacy/autogen client).

User ADCs work correctly, but service account ADCs don't work at all. I haven't tried this on a Google VM yet.

Client

Storage

Environment

 REDACTED  MCW0CDP3YY  ~  Desktop  Git  tpg-2252  ERROR  $  go version
go version go1.15.5 darwin/amd64
 REDACTED  MCW0CDP3YY  ~  Desktop  Git  tpg-2252  $  go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/REDACTED/Library/Caches/go-build"
GOENV="/Users/REDACTED/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/REDACTED/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/REDACTED/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.15.5/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.15.5/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/REDACTED/Desktop/Git/tpg-2252/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/b1/dthn83bs2qbcrg38qszm22440000gn/T/go-build055558340=/tmp/go-build -gno-record-gcc-switches -fno-common"
 REDACTED  MCW0CDP3YY  ~  Desktop  Git  tpg-2252  $  go list -m all | grep go/storage
cloud.google.com/go/storage v1.14.0

Terraform Core uses the new client but TPG uses the legacy client.

PoC:

// Modern client in Terraform Core at backend/remote-state/gcs/backend.go

package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"time"

	"cloud.google.com/go/storage"
	"google.golang.org/api/option"
)

func main() {
	ctx := context.Background()

	// Sets your Google Cloud Platform project ID.
	projectID := os.Getenv("GOOGLE_PROJECT")

	// Creates a client.
	client, err := storage.NewClient(ctx, option.ImpersonateCredentials(os.Getenv("GOOGLE_IMPERSONATE_SERVICE_ACCOUNT")))
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}

	// Sets the name for the new bucket.
	bucketName := "foo" // Intentional, expected API response is 409 not 403 from failed impersonation

	// Creates a Bucket instance.
	bucket := client.Bucket(bucketName)

	// Creates the new bucket.
	ctx, cancel := context.WithTimeout(ctx, time.Second*10)
	defer cancel()
	if err := bucket.Create(ctx, projectID, nil); err != nil {
		log.Fatalf("Failed to create bucket: %v", err)
	}

	fmt.Printf("Bucket %v created.\n", bucketName)

}

 REDACTED  MCW0CDP3YY  ~  Desktop  Git  tpg-2252  $  go run main.go 
2021/03/18 15:47:14 Failed to create bucket: Post "https://storage.googleapis.com/storage/v1/b?alt=json&prettyPrint=false&project=mgcp-1192365-landing-zone": impersonate: status code 403: {
  "error": {
    "code": 403,
    "message": "Request had insufficient authentication scopes.",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",
        "domain": "googleapis.com",
        "metadata": {
          "method": "google.iam.credentials.v1.IAMCredentials.GenerateAccessToken",
          "service": "iamcredentials.googleapis.com"
        }
      }
    ]
  }
}
exit status 1

// Legacy/Autogen Client used in TPG
package main

import (
	"context"
	"fmt"
	"log"
	"os"
	"time"

	"google.golang.org/api/option"
	"google.golang.org/api/storage/v1"
)

func main() {
	ctx := context.Background()

	// Sets your Google Cloud Platform project ID.
	projectID := os.Getenv("GOOGLE_PROJECT")

	// Creates a client.
	client, err := storage.NewService(ctx, option.ImpersonateCredentials(os.Getenv("GOOGLE_IMPERSONATE_SERVICE_ACCOUNT")))
	if err != nil {
		log.Fatalf("Failed to create client: %v", err)
	}

	// Creates a Bucket instance.
	sb := &storage.Bucket{
		Name:     "potato", // Expected Response is 409 and not 403 from failed impersonation
		Location: "us",
	}
	// Creates the new bucket.
	ctx, cancel := context.WithTimeout(ctx, time.Second*10)
	defer cancel()

	if _, err := client.Buckets.Insert(projectID, sb).Do(); err != nil {
		log.Fatalf("Failed to create bucket: %v", err)
	}

	fmt.Printf("Bucket %v created.\n", sb.Name)
}

 REDACTED  MCW0CDP3YY  ~  Desktop  Git  tpg-2252  $  go run main.go 
2021/03/18 16:32:11 Failed to create bucket: googleapi: Error 409: Sorry, that name is not available. Please try a different one., conflict
exit status 1

[codyoss]

I believe this is a duplicate of googleapis/google-api-go-client#731. The workaround for now is to explicitly set your scopes. I would recommend option.WithScopes(storage.ScopeFullControl, "https://www.googleapis.com/auth/cloud-platform"). The long term solution to this issue will be to depend on the new impersonate package being worked on in this PR instead of that experimental client option. If you want to follow up more on this lets move the discussion over to googleapis/google-api-go-client#731.