Bug: Webhook trigger from Bitbucket does not refresh Applications in 2.11.0

[Alwinius]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

After the update from ArgoCD 2.10.9 to 2.11.0 the sync trigger on webhook from Bitbucket does not work anymore.

To Reproduce

1. Install ArgoCD 2.11.0
2. Connect a repository that is hosted in Bitbucket and create Application for that repo
3. Setup a webhook in Bitbucket to trigger ArgoCD https://argocd.cluster.../api/webhook
4. Push to bitbucket
5. Observe in ArgoCD that the application is not refreshed
6. perform a manual refresh to detect changes

Expected behavior

The refresh is triggered automatically.

Version

argocd: v2.11.0+d3f33c0
  BuildDate: 2024-05-07T16:01:41Z
  GitCommit: d3f33c00197e7f1d16f2a73ce1aeced464b07175
  GitTreeState: clean
  GoVersion: go1.21.9
  Compiler: gc
  Platform: linux/amd64

The version of Bitbucket is v8.19.3

Logs

time="2024-05-20T09:21:51Z" level=info msg="Received push event repo: ssh://git@git.domain.com:7999/incs/k8s-aws-eks-staging.git, revision: master, touchedHead: true"             
time="2024-05-20T09:21:51Z" level=info msg="Received push event repo: https://git.domain.com/scm/incs/k8s-aws-eks-staging.git, revision: master, touchedHead: true"                

Bitbucket receives HTTP 200 and the above log is shown in argocd-server

[kyleli666]

I have same issue on v2.11.0 with bitbucket cloud.
I can see "Received push event log" and Bitbucket get 200 response, but No refresh action log in argocd-server.

[mh013370]

Same here.

[kyleli666]

Tested again in v2.10.10 and v2.10.11, the webhook from Bitbucket works.
But v2.11.1 and v2.11.2 still does not work, argocd-server responses code 200 much faster than v2.10.x while not log any app refresh information in the console.
Here is the debug level log from v2.11.2, it seems doing some work after "Received push event repo" but finally not triggring the app refresh.

time="2024-05-24T03:19:16Z" level=info msg="Received push event repo: https://bitbucket.org/myworkspace/mycoderepo, revision: develop, touchedHead: true"
time="2024-05-24T03:19:16Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:19:16Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:19:16Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:19:16Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"

A normal log from v2.10.11 looks like this

time="2024-05-24T03:26:39Z" level=info msg="Received push event repo: https://bitbucket.org/myworkspace/mycoderepo, revision: develop, touchedHead: true"
time="2024-05-24T03:26:39Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:26:39Z" level=info msg="Requested app 'ba-deployment-api' refresh"
time="2024-05-24T03:26:39Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:26:40Z" level=info msg="Requested app 'cb2-conn-test-org-sk3' refresh"
time="2024-05-24T03:26:40Z" level=debug msg="https://bitbucket.org/myworkspace/mycoderepo.git uses repoURL https://bitbucket.org/myworkspace/mycoderepo"
time="2024-05-24T03:26:40Z" level=info msg="Requested app 'cb2-conn-test-org-sk3-jeff01' refresh"

[de-slalonde]

I'm seeing the same issue. On 2.11.0 and 2.11.1 my Bitbucket webhooks do not trigger a refresh. Things are fine on 2.10.9.

[petrlebedev]

2.11.2 Same issue.

[initialed85]

Hi team, I was having this issue using https://artifacthub.io/packages/helm/argo/argo-cd/7.0.0 and it ended up being that I had enabled the ingress for both argocd-server and argocd-applicationsets / Git Generator webhook for the same FQDN.

It seems that they both have handlers for /api/webhook and therefore similarly error patterns, which adds to confusion.

You can check if this is what's happening to you with the following:

kubectl -n argo-cd port-forward service/argo-cd-argocd-server 8080:80

curl -vvv -L -X POST -H 'X-Event-Key: repo:push' -H 'Host: your-real-argo-fqdn.org' http://localhost:8080/api/webhook

Instead of Unknown webhook event you should get Webhook processing failed: event not defined to be parsed

The fix (for me) was to disable that second ingress. I think Kubernetes ingresses merge and I guess argo-applicationsets is defined second (thus overwriting the /api/webhook handler that we need).

Edit:

• https://github.com/argoproj/argo-cd/blob/master/util/webhook/webhook.go#L386
• https://github.com/argoproj/argo-cd/blob/master/applicationset/webhook/webhook.go#L148