Instead of introducing a new key (regex_paths) I think it would be nice to add some consistency to the apis and use StringMatch like in VirtualService and Gateway.
ie:

paths:
- exact: /v1
- regex: "\/.*/docs\/.*"
- prefix: /v2

This feature will be very useful to define the authz policy, without this feature the only way that I can image is to use the Mixer Policy with OPA (it is powerful but seems not a recommend way), so I suggest AuthorizationPolicy to support another header matcher mode (istio/pilot/pkg/security/authz/model/matcher/header.go):

• /exact: for exact string mach
• */suffix: for suffix string match
• prefix/*: for prefix string match
• *: for non-empty string match
• |^/regex/.*$|: for regex string match

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin-1
  namespace: foo

spec:
  selector:
    matchLabels:
      app: httpbin
      version: v1
  rules:
    - from:
        - source:
            principals: ["principal", "principal-prefix-*", "*-suffix-principal", "*"]
            requestPrincipals: ["requestPrincipals", "requestPrincipals-prefix-*", "*-suffix-requestPrincipals", "*"]
            namespaces: ["ns", "ns-prefix-*", "*-ns-suffix", "*"]
            ipBlocks: ["1.2.3.4", "5.6.0.0/16"]
      to:
        - operation:
            methods: ["method", "method-prefix-*", "*-suffix-method", "*"]
            hosts: ["exact.com", "*.suffix.com", "prefix.*", "*"]
            ports: ["80", "90"]
            paths: ["/exact", "/prefix/*", "*/suffix", "*", "|^/regex/.*$|"]

This becomes important in Istio 1.5 now that the alpha Authentication Policy is being replaced with the Request Authentication and Peer Authentication. There is no other way to exclude paths for JWT then to use an Authorization Policy which does not allow regex.

Hmm indeed this seems a feature regression... I confirmed we don't have regex in the beta API (both req/authz). @yangminzhu you must know more on this. We do not include regex in the authz api is that for a specific reason or due to our negligence?

yes, we dropped the regex support in the beta RequestAuthentication and didn't add it in the AuthorizationPolicy as we're not sure how many users will actually need the regex and what's the API should be.

Instead of introducing a new key (regex_paths) I think it would be nice to add some consistency to the apis and use StringMatch like in VirtualService and Gateway.
ie:

paths:

• exact: /v1
• regex: "/./docs/."
• prefix: /v2

@jpatters thanks for the suggestion, we ever considered this a little bit but @liminw has some concerns that this will make the policy quite redudant.

/exact: for exact string mach
/suffix: for suffix string match
prefix/: for prefix string match
: for non-empty string match
|^/regex/.$|: for regex string match

@edgel this seems a more easier and incremental way to support the regex, but we need to think about the detail syntax.

@diemtvu @liminw It looks like many users are actually wanting the regex match in authZ policy, could you share some thoughts here, maybe we could support it like this: paths: ["/exact", "/prefix*", "*-suffix", "*", "regex(...)"]?

We usually don't make a feature request as a P0. Previsouly this was an experimental feature and we don't expect users to use it in production.

+1
Knative on Istio have to use triggerRules.excludedPaths.{exact,prefix} for authentication policy as knative service gets health checking and metrics collection from external(knative system pods). So this feature is really necessary.
Cloud Run's solution also suggested it here for now.

use triggerRules.excludedPaths.{exact,prefix} for authentication policy

new authorization policy has not_paths since 1.5 for negative match @nak3

We usually don't make a feature request as a P0. Previsouly this was an experimental feature and we don't expect users to use it in production.

@myidpt

What is the recommended solution for the following REST pattern of sub-resources?

Suppose I build a CRUD REST service for Countries. Each country has a sub-resource City. Following REST convention, I believe this means paths like
/countries
/countries/{country id}
/countries/{country id}/cities
/countries/{country id}/cities/{city id}

I have a rule to ALLOW requests to /countries/MyCountry* if request.auth.claims["countries"] contains "MyCountry".

Some users who are authorized for /countries/MyCountry* are NOT authorized to POST cities (they are read-only users). My natural thought is to create a rule to DENY POST for /countries/*/cities/* when request.auth.claims["readonly"] is true.

I am not an expert REST developer but I had thought this was an extremely common pattern--can you help me understand why this pattern is uncommon (what are the better patterns for subresources?), or what led to the conclusion that users would not use this regex support for this purpose in production?

I am not an expert REST developer but I had thought this was an extremely common pattern--can you help me understand why this pattern is uncommon (what are the better patterns for subresources?), or what led to the conclusion that users would not use this regex support for this purpose in production?

@lgarrett-isp

Can't speak for everyone, but it's definitely the same issue for us. We've followed what we believe is the correct REST convention, and find ourselves unable to express this within the policy. We're trying to replace our custom Envoy-based sidecar with a full Istio implementation, but the lack of this support means we'll likely end up with some kind of hybrid, which is unfortunate. We definitely need this for Production use-cases.

Yep, we have run head-first into the same; we are moving away from Istio for our REST services until it has more support for basic REST conventions.

@yangminzhu is this solved by #27984?

@yangminzhu is this solved by #27984?

yes, but we're evaluating if glob matching would be good enough for most use cases.

@wereeder @jpatters @edgel @nak3 @lgarrett-isp @DanSalt @ajit-hybris @aguromano @ajitchahal @mcieplak @rolandkool

I'm wondering would the glob matching good enough for your use cases? It looks like most examples (e.g. \/.*swagger.* or /countries/*/cities/*) posted here can be supported by glob matching. I feel there should be some use cases that requires the full regex but not sure detail examples, do you have any examples for this? Or you think the glob matching is indeed good enough? Thanks

@yangminzhu For My header is be like x-auth-request-groups : 'CN=EMSUDEMY,OU=Groups,O=cco.xxx.com, CN=all-xxxx-people,OU=xxxx Groups,O=cco.xxxx.com, CN=xxxxxxxxx,OU=xxxx Groups,O=cco.xxxx.com . I would like match *all-xxxx-people*
Glob matching should be fine for my usecase.

there seems a discussion about this in the community concerning about the security risk @yangminzhu do you have any updates for this regex supporting?

Is there any update for this issue? Will you support glob matching?

+1

is there support for "#" in the paths? I have made a couple of tests, but it does not seem to work. Any help?

1. Please do not comment irrelevant comments on busy issues like this; use https://github.com/istio/istio/discussions
2. # should not be a part of any requests that get to Envoy, per HTTP spec. These are a client side feature of URL only. So authz policy against # does not make sense. https://datatracker.ietf.org/doc/html/rfc3986#section-3.5

this is supported upstream now:
envoyproxy/envoy#31447

Seems like we can expect it to be released in Envoy 1.30. Any idea what the target release date is for that? I'm not sure istio/api#2173 is the right API though given that upstream Envoy just added a new matcher

@keithmattix According to schedule planned release date of Envoy 1.30 is 16/04/2024 - Envoy Major release schedule

Well that link's going in my bookmarks :) Thanks! I think that means that Istio 1.22 can very easily include this feature. Either me or someone on my team can tackle this if @mstrYoda is ok with it? I see they're currently assigned to the issue

Well that link's going in my bookmarks :) Thanks! I think that means that Istio 1.22 can very easily include this feature. Either me or someone on my team can tackle this if @mstrYoda is ok with it? I see they're currently assigned to the issue

It is okay to me, also I would like to contribute too if there is a chance for it 👍

Hi @mstrYoda! I am excited to takle this work together! Here are my thoughts for a solution and how we can pair.

Let's leave the paths and notPaths fields in the AuthorizationPolicy resource as is for backwards compatibility. We can create new fields called pathTemplates and notPathTemplates that can be an array of paths based on uri template matching rules supported by envoy. The user can set one of paths or pathTemplates but not both. If the user wishes to set notPathTemplates they must not set paths or pathTemplates and vise versa. The pathTemplates fields could look something like this:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin
  namespace: foo
spec:
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/default/sa/sleep"]
    - source:
        namespaces: ["test"]
    to:
    - operation:
        methods: ["GET"]
        pathTemplates: ["/info/*/{file}", "/info/v2/*/{file}]
    - operation:
        methods: ["POST"]
        pathTemplates: ["/data/**"]
    when:
    - key: request.auth.claims[iss]
      values: ["https://accounts.google.com"]

/info/** would match /info/foo/bar.txt or info/v2/foo/bar.txt
/data/{file} would match /data/foo/bar/test

Would you be interested in taking on the api changes? I can work on the control plane changes. Let me know your thoughts!

/data/{file} would match /data/foo/bar/test

Just checking this is a typo? :) {name} matches "one path segment up to the next path separator" according to the uri template doc.

I suppose it'd be more effort to only partially support the uri template syntax (i.e. not the named variables) than just passing it straight through to the Envoy config, but as a user I'm wondering if there'd be any value in the named variables beyond it being a kind of documentation: might the captured values get exposed somewhere for (say) use in access logs, or an EnvoyFilter, in a when condition? (If not now, perhaps in future.) And would there be any performance penalty for that over the non-capturing matchers?

Hi @mstrYoda! I am excited to takle this work together! Here are my thoughts for a solution and how we can pair.

Let's leave the paths and notPaths fields in the AuthorizationPolicy resource as is for backwards compatibility. We can create new fields called pathTemplates and notPathTemplates that can be an array of paths based on uri template matching rules supported by envoy. The user can set one of paths or pathTemplates but not both. If the user wishes to set notPathTemplates they must not set paths or pathTemplates and vise versa. The pathTemplates fields could look something like this:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin
  namespace: foo
spec:
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/default/sa/sleep"]
    - source:
        namespaces: ["test"]
    to:
    - operation:
        methods: ["GET"]
        pathTemplates: ["/info/*/{file}", "/info/v2/*/{file}]
    - operation:
        methods: ["POST"]
        pathTemplates: ["/data/**"]
    when:
    - key: request.auth.claims[iss]
      values: ["https://accounts.google.com"]

/info/** would match /info/foo/bar.txt or info/v2/foo/bar.txt /data/{file} would match /data/foo/bar/test

Would you be interested in taking on the api changes? I can work on the control plane changes. Let me know your thoughts!

Hi @jaellio 👋 Thanks for the collaboration. I have the same questions with the @michaelbannister 🤔

I did not work on API side before but I have some idea and experience on the control plane side. Can we pair/collaborate on both? I can make the changes on the API side and would love to see developments on the control plane too.

I am a bit worried about having multiple ways to represent path, especially when its also in 'when'. It can be pretty complex to have 4 (with 'not' variants) rules for the same thing.

I also don't think we need to support the full syntax of Envoy's URI template. The URI template is the implementation detail of our API; we should define the API we want and map it onto the URI template. Specifically, I don't see a strong need to support {name} and *. We should support only one. {} are not valid in URLs, while * is, so in theory that may be more backwards compatible. That also is more future proof if we ever want to make assertions based on the extracted values (I don't think this is supported in Envoy, but its why its in the Envoy API I assume).

/data/{file} would match /data/foo/bar/test

Just checking this is a typo? :) {name} matches "one path segment up to the next path separator" according to the uri template doc.

I suppose it'd be more effort to only partially support the uri template syntax (i.e. not the named variables) than just passing it straight through to the Envoy config, but as a user I'm wondering if there'd be any value in the named variables beyond it being a kind of documentation: might the captured values get exposed somewhere for (say) use in access logs, or an EnvoyFilter, in a when condition? (If not now, perhaps in future.) And would there be any performance penalty for that over the non-capturing matchers?

@michaelbannister Yes, apologies for the typo! I forgot to update the comment :) It should be "/info/*/{file} would match /info/foo/bar.txt and /data/** would match /data/foo/bar/test"

I think for now it would make sense to not support named variables. I had originally assumed that you could reference the variable later on in the template or as an environment variable. Primarily, the variable seems to be a source of documentation (and confusion). We could revisit adding support for named variables later on based on user feedback and increased functionality support (either by Istio or Envoy). For now, we can interpret any paths containing { as invalid.

I am a bit worried about having multiple ways to represent path, especially when its also in 'when'. It can be pretty complex to have 4 (with 'not' variants) rules for the same thing.

@howardjohn Are paths and notPaths supported values for the when conditional?

If we opted to support URI template matching for the existing paths and notPaths fields we would need to interpret * as a templating type and as a valid character in the path. Need to think more on this, but I could see this causing some issues.

@howardjohn Are paths and notPaths supported values for the when conditional?

Ah I guess not, my bad.

I agree * could be a valid value today so that is tricky (although, its a very weird URL). {} are explicitly not valid URL charecters so no one can legitimately have those which may be a good motivation to use them; it also is consistent if we later decide to have {name}

If we only support {} for templating we can use the existing paths and notPaths fields.

If we don't want to incorporate named variables (regardless of whether or not we do anything with them) we could support the following templating and map them to envoy's supported uri templating:

• {} or {*} : Matching one path segment up to the next path separator: /.
• {**} : Matching zero or more path segments. NOTE: It isn't clear from the envoy uri templating documentation if {**} must be the last operator

We will also support the existing * functionality:

• * : Matching zero or many path segments.

Alternatively, if we are interested in incorporating a named variable (but not doing anything with the variable at this time) we could support the same templating with { as described in the uri templating documentation.

I find the named variables introduce unnecessary confusion and would opt for the former option. We can add support for named variables later if necessary.

We will also support the existing * functionality:

• * : Matching zero or many path segments.

Isn't this "Match any character"? Its not "path aware", just equivalent to .* regex. While a {} is more like \/.+?\/ (probably that is not a robust regex, just an example that is close).

I find the named variables introduce unnecessary confusion and would opt for the former option. We can add support for named variables later if necessary.

+1. I would, similarly, not support {*} or {**}. Just {}.

Isn't this "Match any character"? Its not "path aware", just equivalent to .* regex. While a {} is more like /.+?/ (probably that is not a robust regex, just an example that is close).

I think in the UriTemplate context, * becomes path aware (which is one reason why I'm more in favor of @jaellio's original [not]pathTemplate approach).

+1. I would, similarly, not support {*} or {**}. Just {}.

I think the OP's original example would require something like {*}/springfox-swagger-ui/{**}

I would agree * is not path aware currently for Istio.

Depending on how you are defining {} I don't think it supports a complex enough set of use cases.

We could interpret any path with only * and no { as a string matcher. Any path with { would be interpreted as a uri template. We could update all references of {*} to * and all references of {**} to ** (remove all { and }) so the paths would be correctly interpreted by envoy as uri templates.

I think adding new fields (pathTemplates and notPathTemplates) to support more complex templating is simpler if we want to address the asks of the OP: there are less backwards combability concerns, it's more extensible, it'd be easier for users to distinguish between new vs old functionality, and it doesn't feel like we are boxing ourselves into limited templating capabilities. I think our templating support should be scoped, but by choice, not by design or backwards compatibility limitations.

If we aren't in favor of adding new fields, I think we should add support for the following. I don't think {} alone covers enough use cases.

• {} or {*} : Matching one path segment up to the next path separator: /.
• {**} : Matching zero or more path segments. This must be the last operator.

If the path contains { or } then we will translate the path as a uri template rather than as a string match.

This sounds reasonable to me. It'd be good if the corresponding Istio docs were clearer than the UriTemplate docs, though:

* : Matches a single path component, up to the next path separator: /

but then this example:

/videos/*/*/*.m4s would match videos/123414/hls/1080p5000_00001.m4s

which suggests that the . of the file extension can be a boundary of a * as well – but I suspect /videos/*/*/* would also match that same path so maybe it's flexible?

Another thought: Would we want to explicitly disallow any use of { or } which doesn't match one of those specific patterns? e.g. /orders/{id} should fail validation

This sounds reasonable to me. It'd be good if the corresponding Istio docs were clearer than the UriTemplate docs, though:

* : Matches a single path component, up to the next path separator: /

but then this example:

/videos/*/*/*.m4s would match videos/123414/hls/1080p5000_00001.m4s

which suggests that the . of the file extension can be a boundary of a * as well – but I suspect /videos/*/*/* would also match that same path so maybe it's flexible?

Totally agree. This could be a lot more clear - I can verify this by updating the unit tests. I already make some updates to verify if {**} had to be the last operator. envoyproxy/envoy#33104

Another thought: Would we want to explicitly disallow any use of { or } which doesn't match one of those specific patterns? e.g. /orders/{id} should fail validation

I think that explicitly disallowing alternative use cases might be better than an undefined outcome even if we document it as unsupported functionality. I am not sure how complicated the conditional/regex would be, but I think it makes sense to only allow {*} and {**}. If we chose to add support for named variables in the future, we could remove the restriction on our side.

Regarding the impact of explicitly disallowing the use of { if it doesn't match a supported pattern: Previously, if a user provided a path with { or any other combination of brackets the path would technically be invalid but wouldn't result in an error. The path would instead result in no matches since { is not a valid character in a path. By explicitly disallowing unsupported patterns containing { the user will now see an error. I am not sure if this would result in the AuthorizationPolicy not being applied, but this could unintentionally cause authorization policies to not be applied if they contain unsupported paths. Not sure if this is a realistic scenario or how many users this might impact, just wanted to call it out.

Added a comment on the PR - not against adding matches, but only if we understand:

• if HttpRoute can be extended to do the same
• what are the security implications - if routes and authorizations are not aligned ( the API is supposed to work as a 'vendor extension' when using HttpRoute / Gateway )
• backward compatibility risk - now that everything is v1, I don't think we can play games or start moving to v2.

Closing in favor of creating a new issue to track broader adoption and integration of URI templating.