Extreme log spam when Progressive Sync is enabled

[jaredhancock31]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When enabling Progressive Sync, the applicationset-controller spams logs even if the AppSet only contains 2 elements. The problem is exponentially worse if there are multiple AppSets and/or more than a few elements in the set. I assume this is because there's no delay in the for-loop.

This is problematic and costly for anyone exporting logs to Grafana/Datadog/etc. But also the spam effectively makes the appSet controller logs useless from a triage/debugging standpoint.

To Reproduce

Enable Progressive Sync on the appset controller, example helm values:

        configs:
          params:
            applicationsetcontroller.enable.progressive.syncs: true

Create some arbitrary ApplicationSet that uses RollingSync strategy:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: weee
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  goTemplate: true
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: batched
              operator: In
              values: ["true"]
          maxUpdate: 50%
  generators:
    - list:
        elements:
        - item: "foo"
        - item: "bar"
  template:
    metadata:
      name: '{{.item}}'
      labels:
        batched: "true"
#... so on, so forth

Expected behavior

AppSet logs should still be usable, spammy logs should be pushed to higher log levels.
Or perhaps progressive sync's step list should be evaluated on a delay, configurable or otherwise.

Version

Chart Version: 5.54.0
App Version: v2.9.6

Logs

this section is spammed until sync is totally complete

argocd/argo-cd-argocd-applicationset-controller-67c567dd79-6crhd[applicationset-controller]: time="2024-05-15T20:22:16Z" level=info msg="ApplicationSet weee step list:"
argocd/argo-cd-argocd-applicationset-controller-67c567dd79-6crhd[applicationset-controller]: time="2024-05-15T20:22:16Z" level=info msg="step 1: [foo bar]"
argocd/argo-cd-argocd-applicationset-controller-67c567dd79-6crhd[applicationset-controller]: time="2024-05-15T20:22:16Z" level=info msg="Application allowed to sync before maxUpdate?: map[foo:true bar:true]"
argocd/argo-cd-argocd-applicationset-controller-67c567dd79-6crhd[applicationset-controller]: time="2024-05-15T20:22:16Z" level=info msg="end reconcile" applicationset=argocd/tenant-set requeueAfter=0s
argocd/argo-cd-argocd-applicationset-controller-67c567dd79-6crhd[applicationset-controller]: time="2024-05-15T20:22:17Z" level=info msg="generated 2 applications" generator="{&ListGenerator{Elements:[]JSON{{[123 34 116 101 110 97 110 116 34 58 34 111 114 103 105 100 45 56 50 49 55 56 48 50 34 125]},{[123 34 116 101 110 97 110 116 34 58 34 111 114 103 105 100 45 56 49 55 53 56 48 53 34 125]},},Template:ApplicationSetTemplate{ApplicationSetTemplateMeta:ApplicationSetTemplateMeta{Name:,Namespace:,Labels:map[string]string{},Annotations:map[string]string{},Finalizers:[],},Spec:ApplicationSpec{Source:nil,Destination:ApplicationDestination{Server:,Namespace:,Name:,},Project:,SyncPolicy:nil,IgnoreDifferences:[]ResourceIgnoreDifferences{},Info:[]Info{},RevisionHistoryLimit:nil,Sources:[]ApplicationSource{},},},ElementsYaml:,} nil nil nil nil nil nil nil nil nil}"