Rollout status is jumping from paused to progressing/failed in certain timeframe.

[wangli1030]

Checklist:

• I've included steps to reproduce the bug.
• I've included the version of argo rollouts.

Describe the bug

When rollout is in b/g paused status for a long time, the status will jump to progressing and failed, and then back to b/g paused/suspended status in certain timeframe(in my case, every 15 mins). When any resource has prune-last hook, it will caused ArgoCD sync failed.

To Reproduce

Deploy a rollout using b/g strategy and deploy a new version but not promote it. Checking the rollout controller log or rollout status, you will see the rollout status is jumping. Also the the update and transition time of progressing status under rollout status is keep refreshing. This is another indicator that status of rollout is jumping.

Expected behavior

The status should stay in b/g paused/suspended.

Screenshots

v1.6.5

Logs

# Paste the logs from the rollout controller

# Logs for the entire controller:
kubectl logs -n argo-rollouts deployment/argo-rollouts

# Logs for a specific rollout:
kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=<ROLLOUTNAME

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[wangli1030]

From docs, it says

ProgressDeadlineExceeded reason will be surfaced in the rollout status.
Note that progress will not be estimated during the time a rollout is paused.
Defaults to 600s
progressDeadlineSeconds: 600

However, from the code this line, it only escapes the abort status but not rollouts paused. Hence the program will fall into this line and rollout return this

"message":"ProgressDeadlineExceeded: ReplicaSet \"example-rollout-765fcddb5f\" has timed out progressing.","phase":"Degraded"}}".

It will happen every time after progressDeadlineSeconds. Should we also escape the paused condition like abort in RolloutTimedOut function?