It is hard to programmatically manage Spec.Listeners

[evankanderson]

Related to #1246; working on Knative Gateway-API support, one change from the Ingress model is that Gateways explicitly delegate TLS to specific namespaces using Spec.Listeners. This is good, as it can prevent name squatting.

We're using (and managing) separate Listeners and ReferenceGrants for each Knative Route when TLS is enabled, to ensure that the HTTPRoutes we automatically create and manage are correctly associated with TLS certificates and generated hostnames in the correct namespace.

Unfortunately, putting this set of controls into a single resource makes it hard to automatically provision (and de-provision) them in a Controller. In particular, without the ability to set annotations on an individual Listener, it's hard to track whether a particular Listener was previously generated by a Knative Service for the following situations:

• Knative base domain name changed, so all Knative Services have a new URL form. (e.g. service.ns.example.com --> service.ns.127.0.0.1.sslip.io)
• Knative Service deleted, should clean up related Listener configuration. This might seem easy with finalizers, but it becomes harder if you have to worry about controller restart/crash while processing the finalizer. With separate Listener objects, you can put a referencing object UID into an annotation to know whether the Listener was created by your resource or another resource.

All of this is probably solve-able on a single Gateway resource at the cost of some extra code (for example, we can keep a map from Route UID -> Listener entries in an annotation), but having the option for a separate Listener object per HTTPRoute in the Gateway namespace would simplify things dramatically. In that case, Listener objects would work a bit like PersistentVolumes in the storage mode, and HTTPRoutes would work a bit like PVCs in terms of attaching to Listeners.

I know there was some concern about increasing the number of required objects for the API (and I'm sensitive to this concern), so one compromise could be to support either a Listener selector or explicit Listener list on the Gateway object? Other ideas also welcome.

[evankanderson]

Another option (related to #1246) would be to provide an explict way of sharing Spec.Addresses so that you could use a one-Gateway-per-Listener pattern (possibly by copying Status.Addresses from the leader Gateway and putting it in the Spec.Addresses of the "follower" Gateways).

I prototyped this out, but it turns out that the Contour implementation doesn't currently support multiple functional Gateways with the same Spec.Addresses.

[howardjohn]

Istio supports it (I think... it used to at least :-) ) if it helps with prototypes.

[evankanderson]

The problem is that this behavior isn't specified as supported in the spec, so if we want to replace the handful of different HTTP proxy integrations in Knative with a single Gateway API implementation (and we do, it would make installation much easier for users), we need this behavior to either be specified (and the existing Contour behavior non-compliant), or we need some other mechanism for doing this.

[youngnick]

As discussed on the other example, I don't know if we can make this a Core expectation, since it requires Gateway merging. I think that we definitely should make Gateway merging an Extended expectation, so we can say "If you support Gateway merging, the following must be true", and ensure that merging Gateways with matching Addresses works.

[mikemorris]

This does feel like it could be worth exploring if it helps performance/management at scale - Listeners already have enough object-specific functionality that they feel somewhat-independent from Gateways, such as the attached/detached status.

[youngnick]

I'm a big -1 on having a separate Listeners object, because it will need the full panoply of the two-way handshake between the Gateway and Listener objects, just like the ones between Gateway and Listener, because you know that as soon as you separate them, people will want them in different namespaces. And that's on top of the ETOOMANYCRDs problem.

[evankanderson]

My thought was that It would be reasonable to require Listeners to be in the same namespace, particularly because the two-way-handshake problem would repeat the issue of bloating the Gateway CRD (this is a fan-in problem).

Alternatively, explicitly requiring merging of Gateways that specify the same Spec.Addresses (rather than "one wins") could achieve the same outcome. This wouldn't prevent some implementation from saying "you can't merge more than 200 Listeners", but it would give projects that want to put more Listeners onto a single DNS entry a supported way to do so.

Having Gateway merging be an extended feature means programmatic users have one of three choices:

1. Only interoperate with some gateway implementations that implement merging
2. Generate new Gateways for each usage, leading to load balancer IP address wastage and DNS configuration challenges
3. Limit the max number of resources based on apiserver storage limits.

[youngnick]

tbh it's always been my dream that the GatewayClass would end up with a status section that describes what extended features are enabled, so that programmatic users can look there.

I'm definitely in favor of merging Gateways with the same Spec.Addresses instead of a separate Listener resource though.

[mikemorris]

As a minor update, with GEP-2162 GatewayClass can now advertise supported features.

[howardjohn]

I would think the most common case is to *not* set an address at all though (and get one assigned), which may make that a bit tricky. I am not sure exactly how this would look in YAML, but I have always envisioned the correct model to have one "parent" gateway, and other gateways can just attach listeners. That way you have one clear thing that represents the underlying infrastructure. Not sure how we would represent that though.

…

On Mon, Jul 11, 2022 at 9:23 PM Nick Young ***@***.***> wrote: tbh it's always been my dream that the GatewayClass would end up with a status section that describes what extended features are enabled, so that programmatic users can look there. I'm definitely in favor of merging Gateways with the same Spec.Addresses instead of a separate Listener resource though. — Reply to this email directly, view it on GitHub <#1248 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXN6RHJFTKPKYJDQSDDVTTXNVANCNFSM52K6BTVQ> . You are receiving this because you commented.Message ID: ***@***.*** .com>

[evankanderson]

I actually started implementing this until I realized Contour didn't support it.

For each new TLS Knative Route, I'd create a new Gateway with TLS settings for just that route and copy the status.addresses list from a "leader gateway" (specified by configuration) to the generated Route-specific Gateway.

I'd be thrilled if I could rely on that pattern working across different Gateway implementations in 0.6 or 0.7.

[youngnick]

I think it's important to think about L4 and L7 Gateways slightly differently. Right now, for Contour and I think other controllers, the L7 Gateway support is actually implemented with an LB Service to get data to flow to the in-cluster proxy. In Contour's case, it can be configured to look at a specific service and pick up the status.LoadBalancer.ingress stanza, so the details can be copied to each Gateway it picks up.

In the future, I'd imagine that the replacement for this would be that you end up with a L7 Gateway implementation dependent on a L4 Gateway implementation, where the L4 Gateway fills the same function as the LB Service, setting up the actual data path, and exposing the address in its status. The L7 Gateway implementation will then need to pick up the address details from the L4 Gateway, and copy it to Its Gateways.

This is kind of how you can have "parent" Gateways for the address use case right now - I just don't know if anyone's actually done it yet.

[dprotaso]

I've created a GEP issue to help address this - #1713