Plugin auth using AWS IAM doesn't work unless I re-save the Jenkins Credential form in the Jenkins UI

[jonkipu]

Jenkins and plugins versions report

Environment

Jenkins: 2.458
OS: Linux - 5.10.215-203.850.amzn2.aarch64
Java: 21.0.3 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
amazon-ecs:1.49
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
asm-api:9.7-33.v4d23ef79fcc8
aws-credentials:231.v08a_59f17d742
aws-java-sdk-ec2:1.12.696-451.v0651a_da_9ca_ec
aws-java-sdk-ecs:1.12.696-451.v0651a_da_9ca_ec
aws-java-sdk-efs:1.12.696-451.v0651a_da_9ca_ec
aws-java-sdk-minimal:1.12.696-451.v0651a_da_9ca_ec
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.77-225.v26ea_c9455fd9
branch-api:2.1169.va_f810c56e895
caffeine-api:3.1.8-133.v17b_1ff2e0599
cloudbees-folder:6.940.v7fa_03b_f14759
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.11.0-109.vfe16c66636eb_
configuration-as-code:1775.v810dc950b_514
credentials:1337.v60b_d7b_c7b_c9f
credentials-binding:677.vdc9d38cb_254d
display-url-api:2.204.vf6fddd8a_8b_e9
durable-task:555.v6802fe0f0b_82
font-awesome-api:6.5.2-1
git:5.2.2
git-client:4.7.0
gson-api:2.10.1-15.v0d99f670e0a_7
hashicorp-vault-plugin:368.v48134f694db_f
instance-identity:185.v303dc7c645f9
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.9-1
job-dsl:1.87
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
json-api:20240303-41.v94e11e6de726
mailer:472.vf7c289a_4b_420
mina-sshd-api-common:2.12.1-101.v85b_e08b_780dd
mina-sshd-api-core:2.12.1-101.v85b_e08b_780dd
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-groovy-lib:704.vc58b_8890a_384
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2198.v41dd8ef6dd56
pipeline-model-definition:2.2198.v41dd8ef6dd56
pipeline-model-extensions:2.2198.v41dd8ef6dd56
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56
plain-credentials:182.v468b_97b_9dcb_8
plugin-util-api:4.1.0
prism-api:1.29.0-15
scm-api:690.vfc8b_54395023
script-security:1336.vf33a_a_9863911
snakeyaml-api:2.2-111.vc6598e30cc65
ssh-credentials:337.v395d2403ccd4
structs:337.v1b_04ea_4df7c8
trilead-api:2.142.v748523a_76693
variant:60.v7290fc0eb_b_cd
workflow-aggregator:596.v8c21c963d92d
workflow-api:1291.v51fd2a_625da_7
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3894.3896.vca_2c931e7935
workflow-durable-task-step:1336.v768003e07199
workflow-job:1415.v4f9c9131248b_
workflow-multibranch:791.v28fb_f74dfca_e
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:907.v6713a_ed8a_573

What Operating System are you using (both controller, and any agents involved in the problem)?

Only controller is involved, running Alpine Linux

Reproduction steps

I'm using AWS IAM auth to auth to my Vault instance (non-cluster)

Using CasC to configure the plugin and credential for it:

unclassified:
  hashicorpVault:
    configuration:
      vaultCredentialId: "vaultAWSCredential"
      vaultUrl: "https://my-vault-instance"
credentials:
  system:
    domainCredentials:
      - credentials:
          - vaultAwsIamCredential:
              id: "vaultAWSCredential"
              scope: GLOBAL

I have enabled all required auth, secret engines and policies on the Vault side.

I have configured a test VaultSSHUsernamePassword credential to try and fetch from vault (a secret I have previously created on vault)

I am using the 'Test Vault Secrets Retrieval' button to test it and am getting this error:

FAILED to retrieve username key:
com.datapipe.jenkins.vault.exception.VaultPluginException: Key username could not be found in path kv/test

I have tried to fetch the credential via a pipeline job but am getting 'Access Denied' error.

The only way I got this to work was to access the vaultAWSCredential credential via the Jenkins UI re-save the form (without changing anything), from that point all secret fetching started to work.

I then re-cycled the vault container and again the errors appeared, I then proceeded to re-save the form again and everything started working, I repeated this several times and have confirmed that this is the only change I made to make this work.

I also started the Vault container with debug logs and it seems that until I re-save the form, and trying to fetch secrets no logs are written on the Vault side in regards to secret fetching, when I re-save the form and try to fetch secrets I do see logs relating to auth:

auth.aws.auth_aws_d6a5be79: submitting caller identity request: endpoint=https://sts.amazonaws.com
identity: creating a new entity...

Another change I noticed is that when trying to fetch secrets before re-saving the form, the Jenkins log shows the following:

com.datapipe.jenkins.vault.credentials.common.VaultHelper getVaultSecret
Retrieving vault secret path=kv/test engineVersion=2
com.datapipe.jenkins.vault.credentials.common.VaultHelper retrieveVaultCredentials
Retrieving vault credential ID : vaultAWSCredential
com.datapipe.jenkins.vault.credentials.AbstractVaultTokenCredentialWithExpiration
Expiration for  is java.util.GregorianCalendar .... 
com.datapipe.jenkins.vault.credentials.AbstractVaultTokenCredentialWithExpiration
Auth token is still valid for policies ''

But when re-saving the form and fetching secrets the logs read as:

com.datapipe.jenkins.vault.credentials.common.VaultHelper getVaultSecret
Retrieving vault secret path=kv/test engineVersion=2
com.datapipe.jenkins.vault.credentials.common.VaultHelper retrieveVaultCredentials
Retrieving vault credential ID : vaultAWSCredential
com.datapipe.jenkins.vault.AwsHelper
Creating GetCallerIdentity request
com.datapipe.jenkins.vault.AwsHelper
Acquiring AWS credentials
com.datapipe.jenkins.vault.AwsHelper
AWS Access Key ID: ...
com.datapipe.jenkins.vault.AwsHelper
Signing GetCallerIdentity request
com.datapipe.jenkins.vault.credentials.common.VaultHelper getVaultSecret
Retrieving vault secret path=kv/test engineVersion=2
com.datapipe.jenkins.vault.credentials.common.VaultHelper retrieveVaultCredentials
Retrieving vault credential ID : vaultAWSCredential

Expected Results

Secret fetching should work out of the box

Actual Results

Errors about secret not being found at path or access denied

Anything else?

No response

Are you interested in contributing a fix?

No response

[jonkipu]

Added some debug logs to the plugin and rebuilt it, seems that this stems from the plugin caching its auth token with Vault, when restarting vault to test I guess the token was still cached (although I saw no logs in Vault to match this), when updating the credentials I guess it must have reset the token's expiry and generated a new token