Improve webhook to reject plugins that leads to invalid configuration (duplication) when created

[programmer04]

Is there an existing issue for this?

• I have searched the existing issues

Does this enhancement require public documentation?

• I have added an Acceptance Criteria item for adding and/or adjusting public documentation (if applicable)

Problem Statement

Kong Gateway does not accept two or more plugins of the same type attached to a single supported entity (Service, Ingress, HTTPRoute, KongConsumer, and KongConsumerGroup). Some effort has already been made in

• Configuration apply error from applying 2 Prometheus plugins against 1 Service fails to be parsed #5602
• Check duplicate plugins in webhooks #5749

to catch it during admission webhook validation for those entities, but it doesn't prevent the case described below.

Let's consider e.g. Ingress with annotation konghq.com/plugins: p1, p2, p3, where plugins p1 and p2 have been already created and have different types. Plugin p3 hasn't been applied yet. Such Ingress is valid. Webhook will check that there are no plugins with duplicated types. Next, apply plugin p3 which has the same type as p2. It won't be rejected by the webhook, but it leads to an invalid config that is not accepted by Kong Gateway (respective events and log entries are generated).

Proposed Solution

Introduce validation for KongPlugin that will reject a new plugin when it's already referenced by some previously created entity (its name is in the annotation konghq.com/plugins) that also has a plugin of the same type already configured.

To do it when a plugin is added all resources of kinds Service, Ingress, HTTPRoute, KongConsumer, and KongConsumerGroup have to be checked for annotation konghq.com/plugins whether it contains the name of the newly created plugin. If so check other referenced plugins for an entity if they have a different type. When the type is duplicated reject adding a plugin entirely.

It needs to be checked directly with K8s API to make it real-time.

Additional information

The proposed solution is pretty heavy in terms of data needing to be fetched from K8s and searched when a plugin is added. The situation that it prevents is rather an edge case. So whether it's worth implementing it's left for further consideration. The whole issue is more like a food for thought.

Acceptance Criteria

• admission webhook rejects plugin that when added will lead to invalid configuration due to duplication