Intermittent issue with format="GOOGLE_APPLICATION_CREDENTIALS{{ .data.private_key_data }}"

[kawsark]

When I use format= in the secret stanza for Variable interpolation, sometimes EnvConsul sets the incorrect value. This problem is fairly easy to reproduce.

In this case, I am reading a dynamic GCP service account key from Vault. Vault returns 3 fields in data (full key is truncated as ):

vault read -format=json gcp/key/$role
{
  "request_id": "5e493ddb-b794-5bf0-9a44-14bdceb1dcdc",
  "lease_id": "gcp/key/viewer/1XXN6zn7CizhXluJPGthrH1v",
  "lease_duration": 50,
  "renewable": true,
  "data": {
    "key_algorithm": "KEY_ALG_RSA_2048",
    "key_type": "TYPE_GOOGLE_CREDENTIALS_FILE",
    "private_key_data": "ewo<BASE64-KEY>"
  },
  "warnings": null
}

The application expects GOOGLE_APPLICATION_CREDENTIALS to contain the value of private_key_data. This can be achieved by specifying: format="GOOGLE_APPLICATION_CREDENTIALS{{ .data.private_key_data }}"
However, sometimes it gets set as KEY_ALG_RSA_2048 or TYPE_GOOGLE_CREDENTIALS_FILE.

Envconsul version

envconsul v0.9.1 (b5e928a7)

Configuration

testgcp1.hcl file

sanitize=true
upcase=true
log_level="debug"
pid_file="/tmp/envconsul.pid"

vault {
# Using VAULT_ADDR
  renew_token = false

  ssl {
    enabled = false
    verify  = false
  }
}
exec {
  command = "/Users/kawsark/code/local/envconsul/app-wrapper.sh"
  splay = "5s"
}

secret {
  format="GOOGLE_APPLICATION_CREDENTIALS{{ .data.private_key_data }}"
  no_prefix = false
  path = "gcp/key/viewer"
}

Command

envconsul -config testgcp1.hcl

Debug output

Provide a link to a GitHub Gist containing the complete debug output by running
with -log-level=debug.
Please see: https://gist.github.com/kawsark/f659cf6644cbac94527ae7f062dbdb63

Expected behavior

GOOGLE_APPLICATION_CREDENTIALS environment variable should be populated with Base64 encoded Google service account. In the Gist below, when it works, GOOGLE_APPLICATION_CREDENTIALS is shown as below (full key is truncated as ):
https://gist.github.com/kawsark/f659cf6644cbac94527ae7f062dbdb63

Loop # 0
Checking for GOOGLE_APPLICATION_CREDENTIALS
ewo<BASE64-KEY>=

Actual behavior

Although it works most of the time, sometimes GOOGLE_APPLICATION_CREDENTIALS environment variable contain a value of KEY_ALG_RSA_2048 or TYPE_GOOGLE_CREDENTIALS_FILE. In the Gist below, when it does not work, GOOGLE_APPLICATION_CREDENTIALS is set to KEY_ALG_RSA_2048 on line 102
https://gist.github.com/kawsark/f659cf6644cbac94527ae7f062dbdb63#file-envconsul-txt-L100

Loop # 0
Checking for GOOGLE_APPLICATION_CREDENTIALS
KEY_ALG_RSA_2048

Steps to reproduce

1. Using a root or admin token, setup a GCP secrets engine as below:

vault login <root-or-admin-token>
export GOOGLE_CLOUD_PROJECT="<project-name>"
export role="viewer"

vault secrets enable gcp
vault secrets tune -default-lease-ttl=50s -max-lease-ttl=60s gcp
vault write gcp/config credentials=@gcp-root-service-account.json
vault write gcp/roleset/$role project="${GOOGLE_CLOUD_PROJECT}" secret_type="service_account_key" bindings=-<<EOF
resource "//cloudresourcemanager.googleapis.com/projects/${GOOGLE_CLOUD_PROJECT}" {
roles = ["roles/viewer"]
}
EOF

2. Create a policy and token for EnvConsul and try generating a dynamic secret:

# Write a policy
cat <<EOF > envconsul.policy
path "gcp/key/$role" {
  capabilities = ["read"]
}
EOF

vault policy write envconsul envconsul.policy
vault token create -policy=envconsul -ttl=24h
export VAULT_TOKEN=<token-from-above>
# Or: 
# export VAULT_TOKEN=$(vault token create -ttl=24h -policy=envconsul | grep 'token.*s\.' | awk '{print $NF}')

3. Download example application and invoke EnvConsul
   Example application script provided here: https://gist.github.com/kawsark/74a87420dd9a3a1d9a0c7d4276d30aa9

wget https://gist.githubusercontent.com/kawsark/74a87420dd9a3a1d9a0c7d4276d30aa9/raw/9b72a1242aa36f3743a883c632d686e0666208c7/app-wrapper.sh
chmod +x app-wrapper.sh
envconsul -config testgcp1.hcl

References

Are there any other GitHub issues (open or closed) that should be linked here?

[eikenb]

Hey @kawsark, thanks for reporting this.

Any tips on how I might reproduce this without GCP access? I don't have it and am working to reproduce it using just the vault dev server. Thanks.

[eikenb]

@kawsark Comparing what you give above to the documented /gcp/key GET API (linked belos) do you see any differences from the examples there..

https://www.vaultproject.io/api-docs/secret/gcp/#generate-secret-iam-service-account-creds-service-account-key

Based on everything I've read so far (docs and code) the only way I can see this happening is if GCP sometimes returns things mis-mapped. The format you list in the example has it using {{ .data.private_key_data }}. That is rendered using the standard Go template lookup, and there is no outstanding bug against it for returning the wrong data for a reference like that.

If you can save/log the contents of that .data structure during the good/bad modes to see if it changes that might help. Thanks.