ipAllowPolicy with source: Remote and x-forwarded-for IP address does not work

[bartwitkowski]

What steps did you take and what happened:
We have httpproxy config with ipAllowPolicy:

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: grafana
spec:
  ingressClassName: management-contour
  virtualhost:
    fqdn: grafana.example.com
    ipAllowPolicy:
      - cidr: a.b.c.d/29
        source: Remote
    tls:
      secretName: grafana-tls-cert
  routes:
    - conditions:
        - prefix: /
      services:
        - name: grafana
          port: 80

What did you expect to happen:
The IP from range a.b.c.d/29 should be allowed but actually it is blocked with "RBAC: access denied" and "enforced denied, matched policy none" in envoy logs.

Anything else you would like to add:
Contour is configured with num-trusted-hops: 1.

envoy config_dump

        "typed_per_filter_config": {
         "envoy.filters.http.rbac": {
          "@type": "type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBACPerRoute",
          "rbac": {
           "rules": {
            "policies": {
             "ip-rules": {
              "permissions": [
               {
                "any": true
               }
              ],
              "principals": [
               {
                "remote_ip": {
                 "address_prefix": "a.b.c.d",
                 "prefix_len": 29
                }
               }
              ]
             }
            }
           }
          }
         }
        }

envoy debug logs:

[2024-04-10 16:20:40.827][18][debug][rbac] [source/extensions/filters/http/rbac/rbac_filter.cc:114] checking request: requestedServerName: grafana.example.com, sourceIP: 10.1.0.231:41624, directRemoteIP: 10.1.0.231:41624, remoteIP: 10.1.0.231:41624,localAddress: 10.1.0.231:8443, ssl: uriSanPeerCertificate: , dnsSanPeerCertificate: , subjectPeerCertificate: ,
headers: 
':authority', 'grafana.example.com'
':path', '/'
':method', 'GET'
':scheme', 'https'
'x-forwarded-proto', 'https'
'x-forwarded-port', '443'
'x-forwarded-for', 'a.b.c.d:53662,10.1.0.231'
'x-original-url', '/'
'x-appgw-trace-id', '5c603c472568ba5a295db9d154fdff8e'
'x-original-host', 'grafana.example.com' 
'user-agent', 'curl/7.68.0' 
'accept', '*/*'
'x-envoy-external-address', '10.1.0.231'
'x-request-id', '608da26a-8017-4418-ae85-b3dc280dec24'
dynamicMetadata:                                                                                                                                                                           
[2024-04-10 16:20:40.827][18][debug][rbac] [source/extensions/filters/http/rbac/rbac_filter.cc:158] **enforced denied, matched policy none**

Environment:

• Contour version: 1.27.1
• Kubernetes version: (use kubectl version): 1.28.3
• Kubernetes installer & version: AKS
• Cloud provider or hardware configuration: AKS
• OS (e.g. from /etc/os-release): Ubuntu 22.04