Unable to generate signedUrl when using WIF under GKE

[mkherlakian]

Environment details

• OS: Ubuntu running on GKE
• Node.js version: 18
• npm version: 9.3
• @google-cloud/storage version: 7.10

Steps to reproduce

1. Have a GKE cluster configured with Workload Identity Federation, and have correct permissions on GCS buckets (storageAdmin binding via IAM account from GKE service account)
2. Within a GKE cluster's pod, run the following code

// Import the Google Cloud client library
const { Storage } = require('@google-cloud/storage');

// Instantiate a storage client
const storage = new Storage();

// The name of your GCS bucket
const bucketName = 'test_bucket';

async function getSignedUrl() {
  try {
      const [url] = await storage
        .bucket(bucketName)
        .file('/path/to/file.png')
        .getSignedUrl({
          version: 'v4',
          action: 'write',
          expires: Date.now() + 1000 * 100,
          contentType: 'image/png',
        });

      return url;

  } catch (error) {
    console.error('Error:', error);
  }
}

// Call the function to list files
getSignedUrl();

Resulting error:

ERROR: Error: Invalid form of account ID test_account.svc.id.goog. Should be [Gaia ID |Email |Unique ID |] of the account
    at Gaxios._request (/home/node/app/node_modules/.pnpm/gaxios@5.1.3_encoding@0.1.13/node_modules/gaxios/build/src/gaxios.js:140:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Compute.requestAsync (/home/node/app/node_modules/.pnpm/google-auth-library@8.9.0_encoding@0.1.13/node_modules/google-auth-library/build/src/auth/oauth2client.js:382:18)
    at async GoogleAuth.signBlob (/home/node/app/node_modules/.pnpm/google-auth-library@8.9.0_encoding@0.1.13/node_modules/google-auth-library/build/src/auth/googleauth.js:707:21)
    at async sign (/home/node/app/node_modules/.pnpm/@google-cloud+storage@6.12.0_encoding@0.1.13/node_modules/@google-cloud/storage/build/src/signer.js:181:35) {
  name: 'SigningError'
}
[Nest] 80777  - 04/26/2024, 2:35:53 PM   ERROR [ExceptionsHandler] Unable to generate signed URL.
Error: Unable to generate signed URL.
    at GoogleStorageService.<anonymous> (/home/node/app/main.js:12741:23)
    at Generator.throw (<anonymous>)
    at rejected (/home/node/app/node_modules/.pnpm/tslib@2.6.2/node_modules/tslib/tslib.js:167:69)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
command terminated with exit code 137

3. Note that other bucket operations work fine (create, list, upload). This seems to specifically affect signedURL generation - possibly related to Seem to be getting a 403 when using getSignedUrl with Workload Identity #1672 and "Failure from metadata server" when using GKE and Workload Identity #2346 which have no solution.

[ddelgrosso1]

Hi @mkherlakian in your description you mention using @google-cloud/storage v7.10. However, in your stack trace it is showing @google-cloud+storage@6.12.0 and google-auth-library@8.9.0. Can you confirm the correct versions?

[mkherlakian]

Hey @ddelgrosso1! Apologies, I did test a few versions. The Stack trace was indeed from 6.12 but 7.10 behaves the same!

[ddelgrosso1]

@mkherlakian could you possibly test with v7.10 and provide me the output of npm ls google-auth-library?

[mkherlakian]

Sure thing @ddelgrosso1 - here goes:

root@tmp-shell:/test# node sign.js
Error generating signed URL: Error: Invalid form of account ID test_account.svc.id.goog. Should be [Gaia ID |Email |Unique ID |] of the account
    at Gaxios._request (/test/node_modules/gaxios/build/src/gaxios.js:136:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Compute.requestAsync (/test/node_modules/google-auth-library/build/src/auth/oauth2client.js:408:18)
    at async GoogleAuth.signBlob (/test/node_modules/google-auth-library/build/src/auth/googleauth.js:791:21)
    at async sign (/test/node_modules/@google-cloud/storage/build/cjs/src/signer.js:219:35) {
  name: 'SigningError'
}

BTW I did replace test_account in the output, the account we're using is a combination of letters and numbers, and one hyphen, in case that makes a difference.

root@tmp-shell:/test# npm ls google-auth-library
test@ /test
`-- @google-cloud/storage@7.10.2
  `-- google-auth-library@9.9.0

[ddelgrosso1]

Just an update, I setup a GKE cluster today and was able to recreate the same error. I'm still trying to figure out if it is a configuration problem with WIF / GKE (full disclosure I'm not a GKE expert by any means) or a problem in the auth library. Will update once I get to the root cause.

[mkherlakian]

Thanks for the update! It's good that it's reproducible. Yeah the strange thing is that other operations seem to work fine. I guess the mechanism for signed url is possibly different?... Let me know if I can help!

[ddelgrosso1]

So I got this working today. Turns out some adjustments are needed on both the setup / configuration of GKE and on the code side. I'm going to link to this repo which is not google owned but does a much better job walking through the relevant setup than I can. The relevant Node bits are here.

Edit: I'm going to take an action item to discuss this internally and see what we can do about better documenting this officially.

[mkherlakian]

Thanks for taking a look at this @ddelgrosso1. Just went through the repos you linked, what they're doing does make sense.

Is there no way to transparently support this from the library, without having to create a user impersonation? It does add quite a bit of friction to the process, and might lead users to just inject a GOOGLE_APPLICATION_CREDENTIALS instead to avoid it, which is less safe...

[ddelgrosso1]

@mkherlakian fair question. From the perspective of the storage library I'm not sure what, if anything could be done to make this more transparent. This library hands the actual signing call off to the authentication library which in turn handles the metadata calls.

I think the larger gap here isn't necessarily the way storage or auth behaves but rather the docs don't clearly illustrate how to get this properly setup. The auth library has some samples on using the Impersonated client but there is still a lack of end to end setup documentation.

[ddelgrosso1]

@mkherlakian curious if you were able to get this to work?

[ddelgrosso1]

Going to close this out. If there are more questions or problems, feel free to reopen or start a new issue.