AKS' Admission Enforcer mutates ControlPlane's ValidatingWebhookConfiguration which causes a perpetual reconciliation loop

[pmalek]

Current Behavior

When KGO is running against an AKS cluster, that cluster's Admission Enforcer patches all ValidatingWebhookConfigurations so that it does not check AKS managed resources, which have the kubernetes.azure.com/managedby=aks label set.

This is in conflict with KGO's mechanism to patch the in cluster resource if it's different than the one that's generated.

  []v1.ValidatingWebhook{
  	{
  		... // 3 identical fields
  		FailurePolicy: &"Ignore",
  		MatchPolicy:   &"Equivalent",
  		NamespaceSelector: &v1.LabelSelector{
  			MatchLabels: nil,
- 			MatchExpressions: []v1.LabelSelectorRequirement{
- 				{
- 					Key:      "kubernetes.azure.com/managedby",
- 					Operator: "NotIn",
- 					Values:   []string{"aks"},
- 				},
- 				{Key: "control-plane", Operator: "NotIn", Values: []string{"true"}},
- 			},
+ 			MatchExpressions: nil,
  		},
  		ObjectSelector: &{},
  		SideEffects:    &"None",
  		... // 3 identical fields
  	},
...

Expected Behavior

Reconciliation succeeds.

Proposed solutions

• Disable AKS' Admission Enforcer (currently not possible but [Feature] remove the admission enforcer, or allow to be disabled at cluster level Azure/AKS#4002 tracks this)
• Make KGO aware of the running platform ( tracked internally under https://github.com/Kong/gateway-operator-archive/issues/867 ) and account for that when generating ValidatingWebhookConfiguration
• Suggest users to add 3rd party tooling to remove the aks related VWC config in MatchExpressions added by AKS (something akin to Kyverno mutate rules https://kyverno.io/docs/writing-policies/mutate/)
  • Probably won't work as those controllers (AKS and mutating controller) will fight with each other.

Steps To Reproduce

1. Run KGO against an AKS cluster (set logs to trace log level using helm's arg: --set env.zap_log_level=2)
2. Deploy Gateway (e.g. using https://github.com/Kong/gateway-operator/blob/36c58ab4dd9a449627e14381cf1fc63f362b9903/config/samples/gateway-with-gatewayconfiguration.yaml). 2 Gateways make it more apparent, not sure 100% why that's the case.
3. Observe ControlPlane not getting a Deployment and perpetual reconciliation of ValidatingWebhookConfiguration

Operator Version

1.2.3 and latest main

kubectl version