Preflight fail route not found

[root-io]

curl -H "Origin: https://example.com" \
  -H "Host: api.example.com" \
  -H "Access-Control-Request-Method: POST" \
  -X OPTIONS -v -s -L \
  http://$GATEWAY_HOST \
  1> /dev/null
*   Trying xxx.xxx.xxx.xxx:80...
* Connected to xxx.xxx.xxx.xxx (xxx.xxx.xxx.xxx) port 80 (#0)
> OPTIONS / HTTP/1.1
> Host: api.example.com
> User-Agent: curl/8.1.2
> Accept: */*
> Origin: https://example.com
> Access-Control-Request-Method: POST
>
< HTTP/1.1 302 Found
< location: https://api.example.com:443/
< date: Sun, 24 Dec 2023 00:13:04 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host xxx.xxx.xxx.xxx left intact
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://api.example.com:443/'
*   Trying xxx.xxx.xxx.xxx:443...
* Connected to api.example.com (xxx.xxx.xxx.xxx) port 443 (#1)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
} [319 bytes data]
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* (304) (IN), TLS handshake, Unknown (8):
{ [15 bytes data]
* (304) (IN), TLS handshake, Certificate (11):
{ [4031 bytes data]
* (304) (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* (304) (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* (304) (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=example.com
*  start date: Dec 23 19:33:02 2023 GMT
*  expire date: Mar 22 19:33:01 2024 GMT
*  subjectAltName: host "api.example.com" matched cert's "api.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: OPTIONS]
* h2 [:scheme: https]
* h2 [:authority: api.example.com]
* h2 [:path: /]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* h2 [origin: https://example.com]
* h2 [access-control-request-method: POST]
* Using Stream ID: 1 (easy handle 0x126012e00)
> OPTIONS / HTTP/2
> Host: api.example.com
> User-Agent: curl/8.1.2
> Accept: */*
> Origin: https://example.com
> Access-Control-Request-Method: POST
>
< HTTP/2 204
< access-control-allow-origin: https://example.com
< vary: Origin, Access-Control-Request-Headers
< access-control-allow-credentials: true
< access-control-allow-methods: GET,HEAD,PUT,PATCH,POST,DELETE
< date: Sun, 24 Dec 2023 00:13:04 GMT
< x-envoy-upstream-service-time: 4
< server: envoy
<
{ [0 bytes data]
* Connection #1 to host api.example.com left intact

I do not have a cors SecurityPolicy set, access-control-allow-* headers are returned by my backend.

When using my browser, preflight and cors fail 99% of the time (the 1% remaining it works) with this log from eg:

{
  "start_time": "2023-12-24T00:32:48.926Z",
  "method": "OPTIONS",
  "x-envoy-origin-path": "/endpoint",
  "protocol": "HTTP/2",
  "response_code": "404",
  "response_flags": "NR",
  "response_code_details": "route_not_found",
  "connection_termination_details": "-",
  "upstream_transport_failure_reason": "-",
  "bytes_received": "0",
  "bytes_sent": "0",
  "duration": "0",
  "x-envoy-upstream-service-time": "-",
  "x-forwarded-for": "xxx.xxx.xxx.xxx",
  "user-agent": "xxx",
  "x-request-id": "xxx",
  ":authority": "api.example.com",
  "upstream_host": "-",
  "upstream_cluster": "-",
  "upstream_local_address": "-",
  "downstream_local_address": "xxx.xxx.xxx.xxx:10443",
  "downstream_remote_address": "xxx.xxx.xxx.xxx:48762",
  "requested_server_name": "example.com",
  "route_name": "-"
}

The route does exist, I can POST to /endpoint using Postman and it works.

Envoy Gateway latest based on a06377d

[zhaohuabing]

@root-io Does it work if you create a SecurityPolicy with cors?

I suspect the origin of requests from web browser is not "https://example.com". Can you check with chrome -> right button -> inspect -> network tab ?

[root-io]

The origin is correct, I tried with the following SecurityPolicy without success:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: cors
spec:
  cors:
    allowMethods:
    - GET
    - HEAD
    - PUT
    - PATCH
    - POST
    - DELETE
    allowOrigins:
    - type: RegularExpression
      value: example\.com
    allowCredentials: true
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend

I discoverd it does not work with Chrome and Firefox but works with Safari, I don't get it.

Edit: In the Chrome console I get the following error:

Access to fetch at 'https://api.example.com/endpoint' from origin 'https://example.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

It's like the server returns a 404 without any cors headers

[arkodg]

you may need to specify value as .*\.example\.com
https://gateway.envoyproxy.io/latest/user/cors/
@zhaohuabing imo we should make this value look more like how hostname is defined in upstream
https://gateway-api.sigs.k8s.io/reference/spec/#gateway.networking.k8s.io%2fv1.Hostname else users will continue to make similar mistakes

[root-io]

Hey guys thanks for your quick answers, I couldn't get it working, I don't know if there is an issue using kustomize with a custom namespace but even with RegularExpression set to .*\.example\.com it does not work at all, I ended up running emissary-ingress, I'll try again once EG v1.0.0 and cert-manager v1.14 (Gateway API v1 support) are released.

[zhaohuabing]

Hey guys thanks for your quick answers, I couldn't get it working, I don't know if there is an issue using kustomize with a custom namespace but even with RegularExpression set to .*.example.com it does not work at all, I ended up running emissary-ingress, I'll try again once EG v1.0.0 and cert-manager v1.14 (Gateway API v1 support) are released.

@root-io tried and works on my machine. I plan to follow @arkodg 's suggestion to use wildcard matching because it's more intuitive and easier to use. You can try it later after that PR gets merged.

#2389

[zhaohuabing]

@arkodg origin is the combination of scheme, hostname, and port. so maybe we can make the host part using wildcard like upstream, and leave the other parts as exact match? It's less flexible than regex, but more aligned with users' existing experience.

for example:
instead of

.*foo\.com:8080

Users can specify the following origins

https://*.foo.com:8080
http://*.foo.com:8080

API will be like

// CORS defines the configuration for Cross-Origin Resource Sharing (CORS).
type CORS struct {
	// AllowOrigins defines the origins that are allowed to make requests.
	// An origin is the combination of scheme, hostname, and port. The hostname
	// can be “precise” which is a domain name without the terminating dot of a 
	// network host (e.g. “foo.example.com”) or “wildcard”, which is a domain 
	// name prefixed with a single wildcard label (e.g. *.example.com).
	// 
	// For example, the following are valid origins:
	// - https://foo.example.com
	// - https://*.example.com	
	// - http://foo.example.com:8080
	// - http://*.example.com:8080
	// +kubebuilder:validation:MinItems=1
	AllowOrigins []string `json:"allowOrigins,omitempty" yaml:"allowOrigins"`
	// AllowMethods defines the methods that are allowed to make requests.
	// +kubebuilder:validation:MinItems=1
	AllowMethods []string `json:"allowMethods,omitempty" yaml:"allowMethods"`
	// AllowHeaders defines the headers that are allowed to be sent with requests.
	AllowHeaders []string `json:"allowHeaders,omitempty" yaml:"allowHeaders,omitempty"`
	// ExposeHeaders defines the headers that can be exposed in the responses.
	ExposeHeaders []string `json:"exposeHeaders,omitempty" yaml:"exposeHeaders,omitempty"`
	// MaxAge defines how long the results of a preflight request can be cached.
	MaxAge *metav1.Duration `json:"maxAge,omitempty" yaml:"maxAge,omitempty"`
	// AllowCredentials indicates whether a request can include user credentials
	// like cookies, authentication headers, or TLS client certificates.
	AllowCredentials *bool `json:"allowCredentials,omitempty" yaml:"allowCredentials,omitempty"`
}

[zhaohuabing]

@root-io Wildcard matching has been supported. Can you try the latest image?

I have verified this on my dev machine.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: cors-example
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  cors:
    allowOrigins:
    - "http://*.foo.com"
    - "http://*.foo.com:80"
    allowMethods:
    - GET
    - POST
    allowHeaders:
    - "x-header-1"
    - "x-header-2"
    exposeHeaders:
    - "x-header-3"
    - "x-header-4"

curl -H "Origin: http://www.foo.com" \
  -H "Host: www.example.com" \
  -H "Access-Control-Request-Method: GET" \
  -X OPTIONS -v -s \
  http://$GATEWAY_HOST \
  1> /dev/null

output

< access-control-allow-origin: http://www.foo.com
< access-control-allow-methods: GET, POST
< access-control-allow-headers: x-header-1, x-header-2
< access-control-max-age: 86400
< access-control-expose-headers: x-header-3, x-header-4

https://gateway.envoyproxy.io/latest/user/cors/