rollout in cycle with enabled rolloutRestartTargets

[yotles]

I've tried to use VaultStaticSecret to sync secrets from vault. And for proper rotating keys in secret I want to trigger deployment after updating secret in Vault. For this purpose I've added RolloutRestartTriggered to VaultStaticSecret. But even if I pinned version for secret kv-v2 my deployment triggered each minutes with logs Rollout restart triggered for {Deployment ***}. If I add for example to description RefreshAfter: 10s, it will be triggered each 10s.
At this time in logs of vault-secret-operator:
DEBUG events Rollout restart triggered for {Deployment ***} {"type": "Normal", "object": {"kind":"VaultStaticSecret","namespace":"***","name":"***","uid":"***","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"***"}, "reason": "RolloutRestartTriggered"} DEBUG events Secret synced {"type": "Normal", "object": {"kind":"VaultStaticSecret","namespace":"***","name":"***","uid":"***","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"***"}, "reason": "SecretRotated"}

To stop this behavior I've found only one solution, configure RefreshAfter for long period, for example 24h, in this way it works as expected, rollout restart happened only once when secret really changed with rotating version.

My VaultStaticSecret:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: ***
  namespace: ***
spec:
  type: kv-v2
  mount: ***
  path: ***
  destination:
    name: ***
    create: true
  version: 4
  vaultAuthRef: ***
  rolloutRestartTargets:
    - kind: Deployment
      name: ***

Expected behavior

RolloutRestartTriggered happened only when I've changed version of secret in VaultStaticSecret.

Environment

• Kubernetes version: v1.24.0
  • Distribution: OpenShift
• vault-secrets-operator version: 0.4.1

[yotles]

@benashz Do you know about this behavior, is it expected?

[benashz]

@benashz Do you know about this behavior, is it expected?

@yotles - would you mind testing with spec.hmacSecretData=true on your VSS CR? I did not see it set in the example you have provide.

[yotles]

@benashz Thank you for replay. As I can see https://github.com/hashicorp/vault-secrets-operator/blob/v0.4.1/chart/crds/secrets.hashicorp.com_vaultstaticsecrets.yaml hmacSecretData=true is default value. But even if I set it explicitly in VaultStaticSecret config and configured for example refreshAfter: 1m still get recreating pod each sync secret within 1 minute.

[bhaku]

@yotles We had the same problem. After updating to version 0.5.0 it went away, but remember to update the CRD!

[yotles]

@bhaku Yep, looks like this problem disappeared after updating to 0.5.1. Much appreciate for you advice!

[jameshwc]

Same issue here. We have deployments being rollout restarted every 30s (which is the refreshAfter value)
Here's the static secret yaml we have:

 spec:
   destination:
     create: true
     name: secretkv
     overwrite: false
   hmacSecretData: true
   mount: ***/
   path: ***
   refreshAfter: 30s
   rolloutRestartTargets:
   - kind: Deployment
     name: ***
   type: kv-v2
   vaultAuthRef: static-auth

Updated to v0.5.1 didn't solve this issue.

[bhaku]

@jameshwc did you remember about the CRD update? They will not be updated when you update the version using helm.

[skeet70]

On version 0.5.2, CRDs have been updated, but

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: argocd-secret
spec:
  vaultAuthRef: argocd
  mount: secret
  type: kv-v2
  path: argocd/argocd-secret
  destination:
    create: true
    overwrite: true
    name: argocd-secret
  rolloutRestartTargets:
  - kind: Deployment
    name: argocd-server

results in argocd-server coming up, working, and once confirmed working VSO triggers another rolling restart. This happens in an infinite ~1 min cycle so no downtime is detected, but we have some 10,000 revisions of the deployment over the last few days.

[benashz]

@skeet70 - if you are using the rollout restart feature with static secrets, we recommend setting spec.hmacSecretData=true, that will ensure that only updated secret data triggers the rollout restart. In your example, the restarts will probably be triggered VSO restarting, since you are not specifying spec.refreshAfter.

[benashz]

Going to close this issue out. If you are still encountering a related problem, please open a new issue linking to this one. Thanks.