storage: Make GoogleAccessID and PrivateKey optional

[frankyn]

Hi,

I noticed that SignedURLs for Go doesn't make use of the service account directly and has a developer provide parameters to generate the key. Specifically: GoogleAccessID and PrivateKey. I don't think they should be removed but made optional if GOOGLE_APPLICATION_CREDENTIALS is provided.

pkey, err := ioutil.ReadFile("my-private-key.pem")
if err != nil {
    // TODO: handle error.
}
url, err := storage.SignedURL("my-bucket", "my-object", &storage.SignedURLOptions{
    GoogleAccessID: "xxx@developer.gserviceaccount.com",
    PrivateKey:     pkey,
    Method:         "GET",
    Expires:        time.Now().Add(48 * time.Hour),
})
if err != nil {
    // TODO: handle error.
}
fmt.Println(url)

[grayside]

The Python and Nodejs client libraries go further than this, facilitating something more akin to (*ObjectHandle) SignedURL().

Both extract the Google Access ID from the authentication library and provide their equivalent to SignBytes.

• Python Sample
• Python generate_signed_url()
• Nodejs Sample
• Nodejs getSignedUrl

[Bankq]

@grayside @frankyn I'm trying to port my code from python to go. Do you know what is the idiomatic way using this sdk to access service account private key and email before this feature gets implemented?

Thanks!

[frankyn]

Hi @Bankq,

The SDK will access the service account private key and email are provided in 3 different ways. For simplicity I'll call out GOOGLE_APPLICATION_CREDENTIALS, more information is document in the Python Auth User Guide.

Set the environment variable GOOGLE_APPLICATION_CREDENTIALS with the path to your service account file.

The generate_signed_url() code will automatically fill in the private key and email parts to generate a signed URL.

from google.cloud import storage

client = storage.Client()
bucket = client.get_bucket('bucket-id-here')
blob = bucket.get_blob('remote/path/to/file.txt')

print(blob.generate_signed_url(expiration=3600))

PLMK if this helps with your porting. Thanks for reaching out!

[Bankq]

Hi @frankyn

Thanks! Python code is very convenient indeed. I implemented the resumable upload signed url with something like

if os.getenv("CLOUD") == "gcp":
        creds = google.auth.compute_engine.IDTokenCredentials(google.auth.transport.requests.Request(), GCS_API_BASE)
    else:
        creds = google.auth.default()[0]
signature = base64.b64encode(creds.sign_bytes(string_to_sign))

so that it can run in both my local machine and Cloud Functions.

Now I'm trying to port this to Go, but having trouble understanding what is the right way to access f service_acount_email. Do I have to parse GOOGLE_APPLICATION_CREDENTIALS myself?

[frankyn]

Whoops, misunderstood the direction! Apologies.
Here's an example of accessing the service account, it's a bit more overhead than the Python's implementation.

imports

import (
	"context"
	"fmt"
	"io"
	"io/ioutil"
	"strings"
	"time"

	"golang.org/x/oauth2/google"

	"cloud.google.com/go/storage"
)

Example code

       jsonKey, err := ioutil.ReadFile("path/to/service-account.json")
	if err != nil {
		return "", fmt.Errorf("cannot read the JSON key file, err: %v", err)
	}

	conf, err := google.JWTConfigFromJSON(jsonKey)
	if err != nil {
		return "", fmt.Errorf("google.JWTConfigFromJSON: %v", err)
	}

	opts := &storage.SignedURLOptions{
		Method: "GET",
		GoogleAccessID: conf.Email,
		PrivateKey:     conf.PrivateKey,
		Expires:        time.Now().Add(15*time.Minute),
	}

	u, err := storage.SignedURL(bucketName, objectName, opts)
	if err != nil {
		return "", fmt.Errorf("Unable to generate a signed URL: %v", err)
	}

I'm testing out the signing without a service account next such as Compute Engine and I believe also GCF. This isn't as clear in Go.

[Bankq]

@frankyn I see. Thanks!

I was hoping to avoid parsing credentials one more time since SDK has loaded already.
An approach like python's one seems really clean, i.e having a cloud.google.com/go/credentials.Signer interface or something similar.

Thank you very much sir!

[frankyn]

@Bankq I agree it would help to have a similar flow to a signer. If you have spare time and would like to contribute that would be very helpful!

Here's the Compute Engine version. I ran it on a Compute Engine instance to verify that it works as expected. The default service account used is [PROJECT_NUMBER]-compute@gdeveloper.gserviceaccount.com and I granted it the necessary roles to read the data in the Cloud Storage bucket roles/storage.objectViewer and to sign the string to sign roles/iam.serviceAccountTokenCreator to the default compute service account. The service account is defined in Cloud documentation on Application Default Credentials.

package main

import (
  "context"
  "fmt"
  "time"

  "cloud.google.com/go/storage"
  "cloud.google.com/go/iam/credentials/apiv1"
  credentialspb "google.golang.org/genproto/googleapis/iam/credentials/v1"
)

const (
  bucketName = "bucket-name"
  objectName = "object"
  serviceAccount = "[PROJECTNUMBER]-compute@developer.gserviceaccount.com"
)

func main() {
  ctx := context.Background()

  c, err := credentials.NewIamCredentialsClient(ctx)
  if err != nil {
     panic(err)
  }

  opts := &storage.SignedURLOptions{
     Method: "GET",
     GoogleAccessID: serviceAccount,
     SignBytes: func(b []byte) ([]byte, error) {
        req := &credentialspb.SignBlobRequest{
            Payload: b,
            Name: serviceAccount,
        }
        resp, err := c.SignBlob(ctx, req)
        if err != nil {
           panic(err)
        }
        return resp.SignedBlob, err
     },
     Expires: time.Now().Add(15*time.Minute),
  }

  u, err := storage.SignedURL(bucketName, objectName, opts)
  if err != nil {
     panic(err)
  }

  fmt.Printf("\"%v\"", u)
}

One issue is the default email address is not auto populated and it could be per documentation. @jadekler, is there a way in Go libraries to get the default service account for Compute Engine? I wasn't able to find one.

[Bankq]

@frankyn I'll try to cut a patch.

is there a way in Go libraries to get the default service account for Compute Engine? I wasn't able to find one.

You summarized my original question very well!

[jeanbza]

@frankyn https://godoc.org/golang.org/x/oauth2/google#FindDefaultCredentials, but there's no way to pull out the email address automagically. cc @broady

[odeke-em]

@jadekler we can perhaps do this automagically like this

package main

import (
	"context"
	"encoding/json"
	"log"

	"golang.org/x/oauth2/google"
)

type CredentialsFile struct {
	ClientEmail  string `json:"client_email"`
	ClientID     string `json:"client_id"`
	PrivateKey   string `json:"private_key"`
	PrivateKeyID string `json:"private_key_id"`
	ProjectID    string `json:"project_id"`
}

func DefaultCredentialsFile(ctx context.Context, scopes ...string) (*CredentialsFile, error) {
	creds, err := google.FindDefaultCredentials(ctx, scopes...)
	if err != nil {
		return nil, err
	}
	cf := new(CredentialsFile)
	if err := json.Unmarshal(creds.JSON, cf); err != nil {
		return nil, err
	}
	return cf, nil
}

func main() {
	ctx := context.Background()
	creds, err := DefaultCredentialsFile(ctx)
	if err != nil {
		log.Fatalf("Failed to find default credentials: %v", err)
	}
	log.Printf("creds: %#v\n", creds)
}

[odeke-em]

I've mailed out CL https://code-review.googlesource.com/c/gocloud/+/42270 please take a look.

[tsutsu]

Is the plan here to just support APPLICATION_DEFAULT_CREDENTIALS, or should something akin to @frankyn's GCE logic be included in the library to support that environment automatically as well?

[frankyn]

@odeke-em's CL https://code-review.googlesource.com/c/gocloud/+/42270 will use ADC sans GCE. That's an open question I have right now. It'd be better to add GCE credential aware logic into the Go Google Auth library. That's where we are right now @tsutsu.

[odeke-em]

Thanks for the pings!

I've done a bit of research and experimentation and on GCE we can get the authorization token say by

curl -i http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token -
L -H "Metadata-Flavor":"Google"
HTTP/1.1 200 OK
Metadata-Flavor: Google
Content-Type: application/json
Date: Wed, 17 Jul 2019 02:04:01 GMT
Server: Metadata Server for VM
Content-Length: 205
X-XSS-Protection: 0
X-Frame-Options: SAMEORIGIN
{"access_token":"<TOKEN>","expires_in":"<PERIOD>","token_type":"Bearer"}

which we can then use as the Token source for the oauth2 client so this doable IMHO.

[odeke-em]

@frankyn in regards to #1130 (comment)

One issue is the default email address is not auto populated and it could be per documentation. @jadekler, is there a way in Go libraries to get the default service account for Compute Engine? I wasn't able to find one.

If you do

curl -i http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email -L -H "Metadata-Flavor":"Google"

it'll return the default email or we can use https://godoc.org/cloud.google.com/go/compute/metadata#Client.NumericProjectID to get the numeric project-id and prefix that to

"[PROJECTNUMBER]-compute@developer.gserviceaccount.com"

as per https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances

[odeke-em]

I've updated the CL accordingly.

[frankyn]

nice! much cleaner @odeke-em!!

[tbpg]

@odeke-em, any status update here?

[odeke-em]

Thanks for the ping @tbpg! Am back at it here, some CLs had stalled for a bit.

[kerneltime]

@jadekler / @tritone Any update on this, the current model leads to unnecessary repetitive code in every repos that initialized SDK one way only to realize additional steps needed for signed URLs.

[tritone]

@kerneltime thanks for the ping, @frankyn and I have started work on this last week but it's been quite complicated due to how the client is set up and different methods of auth that can be used. We'll keep you posted on progress.

[kerneltime]

Yes I am facing it too, I need it to work when someone uses their application default credentials as well as service account when run in K8S. What is the recommendation for such a scenario?

[tritone]

It might be helpful to look at @odeke-em 's PR here: https://code-review.googlesource.com/c/gocloud/+/42270 . This is the starting point for the work that I'm doing but it already contains some logic for handling the ADC as well as service account auth scenarios (see the credentialsFileFromGCE function in storage.go in the PR).

[salrashid123]

@frankyn @tbpg

there are many sources that can sign on behalf of a GCP service account
i've implemented a number of crypto.Signer, crypto.Decrypter n this unofficial repo for GCP:

https://github.com/salrashid123/signer

and in all of those, i cna't derive the required GoogleAccessID

a usage for the PEM-based key is like this... I know, ther'es no real point of the PEM key (i just have it in the repo for testing,etc...you can substitute TPM, KMS or even Vault depending on you need

	r, err := sal.NewPEMCrypto(&sal.PEM{
		PrivatePEMFile: "server.key",
	})
	if err != nil {
		log.Println(err)
		return
	}

	bucket := "mineral-minutia-820-bucket"
	object := "foo.txt"
	keyID := "123456"
	expires := time.Now().Add(time.Minute * 10)

	s, err := storage.SignedURL(bucket, object, &storage.SignedURLOptions{
		Scheme:         storage.SigningSchemeV4,
		GoogleAccessID: keyID,
		SignBytes: func(b []byte) ([]byte, error) {
			sum := sha256.Sum256(b)
			return r.Sign(rand.Reader, sum[:], crypto.SHA256)
		},
		Method:      "PUT",
		Expires:     expires,
		ContentType: "image/png",
	})

---

(i suppose i could also add an iam-based signer as in #1130 (comment) ..that api is actually used to derive a
ImperpsonatedTokenSource

[makuc]

To get default service account (GoogleAccessID) and Project ID:

package main

import (
	"context"
	"log"

	compMeta "cloud.google.com/go/compute/metadata"
	"github.com/makuc/a-novels-backend/pkg/gcp/gcse"
	"golang.org/x/oauth2/google"
)

func Function(ctx context.Context, e GCSEvent) error {
	creds, err := google.FindDefaultCredentials(ctx)
	if err != nil {
		return err
	}
	token, err := creds.TokenSource.Token()
	if err != nil {
		return err
	}
	accountIDRaw := token.Extra("oauth2.google.serviceAccount")
	accountID, ok := accountIDRaw.(string)
	if !ok {
		log.Fatal("error validating accountID")
	}

	client, err := google.DefaultClient(ctx)
	computeClient := compMeta.NewClient(client)
	email, err := computeClient.Email(accountID)
	if err != nil {
		return err
	}
	projectID, err := computeClient.ProjectID()
	if err != nil {
		return err
	}
	log.Printf("Email: %v, ProjectID: %v", email, projectID)

	return nil
}

It gets parsed from compute-metadata, as can be seen when token.Extra("oauth2.google.tokenSource") returns compute-metadata.

Basically, we need to import cloud.google.com/go/compute/metadata, but I aliased it as computeMetadata here because it clashes with cloud.google.com/go/functions/metadata otherwise...

[cee-dub]

I'd love to see this picked up. The CL https://code-review.googlesource.com/c/gocloud/+/42270 looks extremely helpful, but hasn't seen any action in 6 months. I'd love help if possible, but many of the links in the failed integration tests aren't accessible to me.

[MaikuMori]

Have to agree, this one should get some attention.

[tritone]

Thanks for your patience on this, and apologies for the delay. I've been sidetracked with several other projects (including a bunch of other changes that we made to Signed URLs for v4 signature support), but plan on getting back to this shortly.

Where this stands currently is that I tried back in the fall to finish off https://code-review.googlesource.com/c/gocloud/+/42871 , but the approach from that PR (tying the SignedURL method to BucketHandle) proved not workable because of the way that the storage client is designed. I'm instead planning on going with an approach like that of the C# client library, which has a separate URLSigner client that can handle the auth aspects in a sensible way (see https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers#storage-signed-url-object-csharp for what this looks like to use). When a PR for that is available, it'll be linked here.

[jz222]

I was wondering if there is a workaround for Cloud Run. With the help of the above-mentioned workarounds, I managed to get it working locally with the pem key as well as with the json credentials. However, when deploying the container to Cloud Run it fails as it cannot find the required information. Adding the credentials to the container is no option.

creds, err := ioutil.ReadFile(os.Getenv("GOOGLE_APPLICATION_CREDENTIALS"))
if err != nil {
	return storage.PostPolicyV4{}, err
}

conf, err := google.JWTConfigFromJSON(creds)
if err != nil {
	return storage.PostPolicyV4{}, err
}

policy, err := storage.GenerateSignedPostPolicyV4("bucket", "file.png", &storage.PostPolicyV4Options{
	GoogleAccessID: conf.Email,
	PrivateKey:     conf.PrivateKey,
	Expires:        time.Now().Add(5 * time.Minute),
	Conditions: []storage.PostPolicyV4Condition{
		storage.ConditionContentLengthRange(0, 1<<20),
	},
})

Adding the stringified credentials file as an environment variable works, but feels a bit hacky and I was wondering if there was a better way to achieve this.

creds := keys.GetKeys().GOOGLE_APPLICATION_CREDENTIALS_STRINGIFIED

conf, err := google.JWTConfigFromJSON([]byte(creds))
if err != nil {
	return storage.PostPolicyV4{}, err
}

policy, err := storage.GenerateSignedPostPolicyV4("bucket", "file.png", &storage.PostPolicyV4Options{
	GoogleAccessID: conf.Email,
	PrivateKey:     conf.PrivateKey,
	Expires:        time.Now().Add(5 * time.Minute),
	Conditions: []storage.PostPolicyV4Condition{
		storage.ConditionContentLengthRange(0, 1<<20),
	},
})

[codyoss]

Adding the stringified credentials file as an environment variable works, but feels a bit hacky and I was wondering if there was a better way to achieve this.

Have you considered making use of secretmanager to store this info? GetSecret

[salrashid123]

you should be able to grant the service account cloud run uses the tokenCreator role on itself and then utitlize the IAM API as described ealy on here #1130 (comment)

  opts := &storage.SignedURLOptions{
     Method: "GET",
     GoogleAccessID: serviceAccount,
     SignBytes: func(b []byte) ([]byte, error) {
        req := &credentialspb.SignBlobRequest{
            Payload: b,
            Name: serviceAccount,
        }
        resp, err := c.SignBlob(ctx, req)
        if err != nil {
           panic(err)
        }
        return resp.SignedBlob, err
     },
     Expires: time.Now().Add(15*time.Minute),
  }

that way, you're not even dealing with actual raw keys...

@codyoss if you're working on the impersonated credentials stuff, that uses iam.generateAccessToken(), if you want to, you could also wrap the iamAPI as a crypto.Signer() (i.,e expose iam.signBlob()), eg like here...i'll try to put an example of that shortly

[salrashid123]

ok, her'es a sample of the crypto.Signer() thing with iam.signBlob() . IMO, having a full-blown crypto signer is nice for some rather uncommon usecases but in this case, just directly signing with iamcredentials api would be fine (stillno keys)

[derekperkins]

Any chance this will get picked up? This issue has triple the 👍 over the second highest voted issue in this repo.

[ellisonleao]

Any updates on this?

[tritone]

We have a PR up for this at #4604. We were able to figure out a way to make the BucketHandle.SignedURL method approach work and guarantee that the same credentials are used for URL signing as are used for the client that the BucketHandle is tied to; it also covers the GCE use case. Feel free to take a look and weigh in on the PR if you have thoughts!

[derekperkins]

Thanks for getting that merged everyone involved! Can't wait to try it out in the next storage release!

[salrashid123]

+1 thanks, its a common enough ask for GCE|CloudRun|GCF to create signedURLs. Having an easy interface like this is great.(i'm awaiting the new release and i'll then update the sample set for that here