Istio operator fails when adding overlay for MutatingWebhook "istio-revision-tag-default"

[jeswinkoshyninan]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

Since from 1.19.X istio version to current version in the master, AFAIK if we are not setting revision (revision="") is considered as default revision and ending up with two new Webhook configuration called as "istio-revision-tag-default" and "istiod-default-validator". When we are trying to add k8s.overlay to MutatingWebhookConfiguration(at least tested) the operator will fails with following error. And the reason is that rendering of web hook manifest is separated out as per func. Can you please share some insights about this.

2024-04-29T14:40:30.373677Z	error	installer	Error during reconcile: overlay for MutatingWebhookConfiguration:istio-revision-tag-default does not match any object in output manifest. Available objects are:
HorizontalPodAutoscaler:istio-system:istiod
ClusterRole::istiod-clusterrole-istio-system
ClusterRole::istiod-gateway-controller-istio-system
ClusterRoleBinding::istiod-clusterrole-istio-system
ClusterRoleBinding::istiod-gateway-controller-istio-system
ConfigMap:istio-system:istio
Deployment:istio-system:istiod
ConfigMap:istio-system:istio-sidecar-injector
MutatingWebhookConfiguration::istio-sidecar-injector
PodDisruptionBudget:istio-system:istiod
ClusterRole::istio-reader-clusterrole-istio-system
ClusterRoleBinding::istio-reader-clusterrole-istio-system
Role:istio-system:istiod
RoleBinding:istio-system:istiod
Service:istio-system:istiod
ServiceAccount:istio-system:istiod
ValidatingWebhookConfiguration::istio-validator-istio-system
- Pruning removed resourcespanic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x259f44d]

The following command output shows the difference in time in which these webhooks are getting deployed.

➜ ~ kubectl get mutatingwebhookconfiguration             
NAME                    WEBHOOKS  AGE
istio-revision-tag-default         5     2m12s
istio-sidecar-injector           5     2m43s
➜ ~

Why non-revisioned profile ending up with "default" revision ? Please let me know the intention of this change ? In that case, we have to find a way to add overlays to these web hooks.

Version

$istioctl version
client version: 1.19.9
control plane version: 1.19.9
data plane version: 1.19.9
$kubectl version
Client Version: v1.27.4
Server Version: v1.27.4

Additional Information

No response

[jeswinkoshyninan]

I have found that the problem is causing because the processing of default webhooks is not from the here instead it is process after those manifests/resources are getting applied/reconciled. I would assume to keep the processing of default web hook inside of mergeProfile.