Listener/Gateway level extensions

[EItanya]

Warning: This discussion may veer into the territory of extensions vs policy attachment again, but the intent is to narrow the scope specifically around Gateway extensions for the moment.

Summary

The HTTPRouteRule and HTTPRouteBackendRef both support lists of filters which contain extensionRefs. These extensionRefs allow implementations of the Gateway API to provide custom behavior which users can enable on those specific objects by targeting the resource in question. However, right now these extensions are very limited. They only allow attachment in these 2 places. This raises a number of issues in my mind:

1. How does a user apply Gateway level config/extension?
2. How does a user apply default/override config to groups of routes without specifying the extensions on each and every HTTPRouteRule?

The current answer to this question seems to be policy attachment, but there also seems to be appetite for extension points elsewhere in the API as can be seen by @robscott's comment.

Policy attachment seems like the perfect solution to apply policies which cannot have extensions refs due to lack of ownership by the Gateway API community. For example, services. However, the current existence of extension refs imply multiple ways to "skin a cat" which may lead to confusion down the road.

I absolutely understand the power of policy attachment in this case because it allows for attachment at any level without the Gateway API needing to specify extension points at EVERY level of it's API, but this contradiction already exists and so I'm wondering if there's appetite for at least Gateway/Listener level extensionRefs in the meantime to allow consistency of extension between the HTTPRoute and the Gateway.

I'm happy to add examples and use-cases below if this discussion gets some traction.

[howardjohn]

My 2c (which is not aligned with the current API, fwiw), is extensionRefs is a mistake and we should use policy attachment everywhere. Having two ways to attach extensions is a recipe for confusion.

extensionRefs itself isn't so bad (if its everywhere), but with policy attachment its redundant and confusing I feel

[robscott]

This is a good point that we need to consider carefully here. We're stuck in a bit of an impossible position here where I think most would agree with the following statements:

1. It's not viable to add extensionRefs or policyRefs to every resource that could be tied to Gateway API, particularly Service, ServiceImport, Namespace, Pod, and likely others.
2. It's easier to understand references from a Gateway API resource to extensions/policies (forward references) than from extensions/policies to a Gateway API resource (backward references).

Unfortunately those two statements leave us with a tough decision. Either we can choose to use the policy attachment model for everything to provide a more consistent experience, or we can choose to enable forward references from Gateway API wherever we can.

Some points in favor of forward refs:

• Allow reuse of policies/extensions
• Easy to understand which extensions are affecting a resource

Some points in favor of back refs (policy attachment):

• Consistent pattern across all resources
• Allows for different personas to own where/how different policies are applied (impossible when policies are attached by route owner)

IMO, policy attachment is not going anywhere, so the question is if we want to allow this kind of alternative in addition to policy attachment.

It seems at least theoretically viable to me that extensions are distinct from policies. For example, a Route owner may want to attach a CustomAuthFilter, which is clearly something Route owners should be able to control + reuse. On the other hand, it seems possible that a higher level admin may want to control a firewall policy, and that would be a better fit for policy attachment. I'm not sure what the best answer is here, but in any case, this shows that we could benefit from clear guidelines for when to use one approach over the other.

As one final note, this feels closely related to #2593 which proposes adding names to route rules. That would mark the first time where both policy attachment and extension refs could be used to attach to the same place.

[maleck13]

One point in favour of policy attachment I would make, is it allows for external parties (IE not gateway providers) to offer a set of policy and policy enforcement components to solve problems whereas with the current design, extensionRef has to be implemented by the gateway provider. So moving to just support extensionRef would limit the potential for interesting new use cases and communities to grow around gateway API (my biased 2c as Kaudrant is one of these emerging communities)

[youngnick]

I think that's a great point, @maleck13, and I'd argue that the converse is the reason extensionRef exists: those specific points are places where we anticipated that it would be relatively common for implementations to either run ahead of the API (as Envoy Gateway has with Auth filters), or to have their own custom functionality that served as a value add for that implementation.

Maybe a better way to put it is that the current extensionRef fields are pretty tightly bound to an implementation - the number of times when you would implement a Filter that was implemented by some external controller to the main implementation is very small, for example, whereas Policy Attachment doesn't have to be implemented by the main controller.

To be honest, another case that I thought would be more widely used than it seems to be is a matching set of ExtensionRef and Policy - the ExtensionRef opts into a behavior, and then the Policy attaches higher up and sets defaults for instances of the ExtensionRef filter. This would allow, for example, an Auth filter to be specified on a per-route level, with all the repeated settings specified as a Policy to the Gateway. That gives minimal opt-in for the functionality, while allowing Chihiro the Cluster Admin to configure the defaults (or maybe even overrides, since Auth is a security feature) for the Gateway.

All of this isn't to say that I don't see the argument for having only one way of doing things. However, I think that the user experience impact of the Policy Attachment backreference model shouldn't be underestimated.

[EItanya]

the ExtensionRef opts into a behavior, and then the Policy attaches higher up and sets defaults for instances of the ExtensionRef filter

This is similar to the UX we've landed on for the time being, especially while waiting for #2593. #713 offers some interesting ideas for default/override behavior, but I think those need some refining.

[ilackarms]

i agree there should only be one way of doing things. it will quickly get out of hand if we adopt separate conventions across implementations.

one of the things that's nice about having extension refs be inlined inside the Route/Listener/Gateway is that they're able to indicate to the controller the intent to attach a policy by the config author. this prevents ambiguities from arising during translation (for example when a route is present in the controller's cache but the policy is not). this is the major advantage i see to having policy refs be specified directly within the route - the controller is aware when a policy is intended to be attached but is unavailable.

imo combining route delegation (the ability to inherit policies hierarchically) and specifying extension ref at the GW/Route/Listener level handles both the cases: where you want to ensure a config gets applied across a set of routes without each route needing to explicitly opt in, as well as ensuring that routes are never unintentionally served without the intended set of policies that impact how they're served.

there are still benefits to putting the target ref inside of the policy rather than having them on the individual route rules but i think it will be difficult to move in this direction as it provides a completely different way of accomplishing the same config from what's there today.

a middle ground could be achieved where you specify a single 'meta-policy', attach it to an extensionRef placed at the root of the route's spec, and then all the rules from that route table would inherit the same meta policy.

or if we were to decide not to add extensionRef to the root level of the route spec, we can use delegation where you always have a root route that selects a meta policy. the meta policy can then use label selectors to dynamically apply default policies to individual rules written in delegated route tables.

note that in this model it is essential that delegation is able to select child routes via labels, otherwise the system cannot be made self service.

without relying on delegation+references, it will be cumbersome or maybe impossible to use the current api (filters/extension refs on the route rule) in order to enable config defaults and overrides across routes or rules (since using filters, every route would essentially have to opt-in to policy).

i can follow this up with some examples but i wanted to contribute these thoughts to the discussion first

[sanjaypujare]

This discussion seems to imply both policies (or metaresources) and filters are ways to provide custom behavior or extensions. The doc for HTTPRouteFilter clearly states that "HTTPRouteFilter defines processing steps that must be completed during the request or response lifecycle." which means adding custom behavior to request/response processing.

However GEP-713 states that "A Policy Attachment can affect specific settings across ...one object ... or objects in a hierarchy" and does not talk about custom behavior or processing steps. For metaresource it says "A Kubernetes object that augments the behavior of an object in a standard way is called a Metaresource."

I am trying to differentiate between only affecting settings (i.e. properties or attributes) of an object vs affecting behavior or processing steps related to an object such as processing of a request by an HTTPRoute.

So my questions are :

• do policies involve custom behavior or just settings modification?
• are there policies or metaresources which can be used for custom behavior (such as additional or custom processing steps during request processing?)
• if there is custom behavior involved in policy or metaresource attachment how does it interact with filters processing? For example are the processing steps of an attached metaresource executed before any filter processing?
• if there are 2 metaresources attached to an HTTPRoute and both involve custom behavior how are the 2 ordered?
• the processing steps of an attached metaresource are executed only if at least one of the route-rule in an HTTPRoute applies to a request?

[sanjaypujare]

Ideally implementation related limitations should not dictate the design/spec of an API (like the Gateway API) and in this case the limitation is related to how Envoy (an implementation) works.

...What if for instance an implementation does NOT want to allow the user to order the filters because they only want to support specific use cases?

In that case the implementation can reject the specified (listed) order of filters since the implementation cannot support it. So what I am thinking is we specify that the execution order of filters is the listed order and an implementation can reject it if that order cannot be supported.

[EItanya]

Ok I think I see what you're saying. If that's the case then back to your earlier question, how would we enforce that if policies also apply at this level?

[sanjaypujare]

...If that's the case then back to your earlier question, how would we enforce that if policies also apply at this level?

What I am thinking is an attached policy can be logically thought of as inserting a filter at all relevant points in a route (even if the policy does not physically do it to accomplish its task). If the order after such an insertion cannot be supported then the Gateway can indicate some status for the route - pls see my comment #1598 (comment)

[EItanya]

I will also comment on that issue, but I don't think that solves the issue. Taking the example from that issue, let's say you have filter1, filter2, and urlRewrite specified in the HTTPRoute, but filter3 is defined via a policy. At what point in the list is filter3 inserted?

[sanjaypujare]

Good point. I have added a comment with a suggestion in that issue.