How to increase the max header size for services within the mesh via an EnvoyFilter?

[tverghis]

Istio control plane version: 1.18.2
Istio data plane version: 1.18.2

I'm trying to increase the maximum header size acceptable by a service. I've applied the following EnvoyFilter in the service namespace:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: max-header-size-patch
  namespace: istio-test-app
spec:
  workloadSelector:
    labels:
      app: istio-test-app
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: ANY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
          max_request_headers_kb: 96

When I try to send requests to this service with a significantly smaller header size, I still get a 431 Request Header Fields Too Large response.

$ cat /dev/urandom | base64 -w 0 | head -c 30000 > data.txt

$ ls -lh
total 32K
-rw-r--r--    1 root     root       29.3K Apr  4 16:17 data.txt

$ http GET http://istio-test-app-service/anything X-Data:@data.txt
HTTP/1.1 431 Request Header Fields Too Large
content-type: text/plain; charset=utf-8
date: Thu, 04 Apr 2024 16:18:08 GMT
server: istio-envoy
transfer-encoding: chunked
x-envoy-decorator-operation: istio-test-app-service.istio-test-app.svc.cluster.local:80/*
x-envoy-upstream-service-time: 0

431 Request Header Fields Too Large

Conversely, if I reduce the max_request_headers_kb to, say, 1KB, the filter seems to be applied correctly. In that case, requests that would previously succeed with 2KB headers will fail after setting max_request_headers_kb to 1KB. The opposite, increasing the maximum header size, does not seem to work.

Also, the default value is supposedly 60KiB, but I see requests failing with headers far smaller than that - in the ~20KiB range.

[tverghis]

The above EnvoyFilter works fine. The issue we were seeing was due to another component reporting a 431.