Listener not accepting HTTPS without a * host being defined, but with a * host, mappingSelector do not work

[dmaclaury]

Describe the bug
Through testing locally and on EKS 1.29 I have run into this same issue.

• Without a Host defined for the hostname *, a listener defined with protocol: HTTPS does not accept HTTPS/TLS connections
• If I create a host with hostname: "*", mappingSelector does not work on more specific hosts.

To Reproduce
Steps to reproduce the behavior:

1. Deploy Emissary following the getting started guide, and these values:

namespace:
  name: emissary
ingressClassResource:
  name: emissary
service:
  type: ClusterIP
replicaCount: 1
resources:
  limits:
    # cpu: 100m
    memory: 6000Mi
  requests:
    # cpu: 100m
    memory: 3000Mi

helm install emissary-ingress --namespace emissary datawire/emissary-ingress --version 8.9.1 -f emissary-values.yaml 
NAME: emissary-ingress
LAST DEPLOYED: Mon Apr  8 11:02:05 2024
NAMESPACE: emissary
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
-------------------------------------------------------------------------------
  Congratulations! You've successfully installed Emissary Ingress!

-------------------------------------------------------------------------------
To get the IP address of Emissary, run the following commands:
  export POD_NAME=$(kubectl get pods --namespace emissary -l "app=emissary-ingress,release=emissary-ingress" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward $POD_NAME 8080:80

For help, visit our Slack at http://a8r.io/Slack or view the documentation online at https://www.getambassador.io.

2. Apply the qotm test service from getting started guide

kubectl apply -f https://app.getambassador.io/yaml/v2-docs/3.9.1/quickstart/qotm.yaml

deployment.apps/quote created
service/quote created

3. Create a self-signed cert following the steps from https://www.getambassador.io/docs/emissary/latest/howtos/tls-termination

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -subj '/CN=ambassador-cert' -nodes

kubectl create secret tls tls-cert --cert=cert.pem --key=key.pem

secret/tls-cert created

4. Create the following listener configs (and port forward 443 on the service to 10443 locally)

---
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: emissary-ingress-listener-8080
  namespace: emissary
spec:
  port: 8080
  protocol: HTTP
  securityModel: XFP
  hostBinding:
    namespace:
      from: ALL
---
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: emissary-ingress-listener-8443
  namespace: emissary
spec:
  port: 8443
  protocol: HTTPS
  securityModel: XFP
  hostBinding:
    namespace:
      from: ALL

5. Create these hosts

---
apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  name: wildcard-host
spec:
  hostname: "*"
  acmeProvider:
    authority: none
  tlsSecret:
    name: tls-cert
  mappingSelector:
    matchLabels:
      hostKind: wildcard-host
  requestPolicy:
    insecure:
      action: Route
---
apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  name: localhost2
spec:
  hostname: localhost2
  tlsSecret:
    name: tls-cert
  mappingSelector:
    matchLabels:
      hostKind: localhost2
  requestPolicy:
    insecure:
      action: Route
---
apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  name: localhost-splat
spec:
  hostname: "*.localhost"
  tlsSecret:
    name: tls-cert
  mappingSelector:
    matchLabels:
      hostKind: localhost-splat
  requestPolicy:
    insecure:
      action: Route

6. Create these mappings

---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  labels:
    hostKind: wildcard-host
  name: quote-backend-wildcard
  namespace: default
spec:
  docs:
    path: /.ambassador-internal/openapi-docs
  prefix: /backend/
  service: quote
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  labels:
    hostKind: localhost2
  name: quote-backend
  namespace: default
spec:
  docs:
    path: /.ambassador-internal/openapi-docs
  prefix: /backend/
  service: quote
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: quote-backend-host
  labels:
    hostKind: localhost2
spec:
  prefix: /
  service: quote
  docs:
    path: "/.ambassador-internal/openapi-docs"
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: quote-backend-host-splat
  labels:
    hostKind: localhost-splat
spec:
  prefix: /splat-only/
  service: quote
  docs:
    path: "/.ambassador-internal/openapi-docs"

7. Check which mappings work - results attached
   https_with_wildcard_test.txt
   But only the quote-backend-wildcard mapping is effective

curl -kvv https://localhost:10443/backend/
curl -kvv https://localhost:10443/ -H "Host:localhost2"
curl -kvv https://localhost:10443/backend/ -H "Host:localhost2"
curl -kvv https://localhost:10443/backend/ -H "Host:test.localhost"
curl -kvv https://localhost:10443/splat-only/ -H "Host:test.localhost"

8. Delete host/wildcard-host

k delete host/wildcard-host                                           
host.getambassador.io "wildcard-host" deleted

9. Re-test, and we receive SSL errors on the mappings that should complete

curl -kvv https://localhost:10443/ -H "Host:localhost2"
*   Trying [::1]:10443...
* Connected to localhost (::1) port 10443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version
* Closing connection
curl: (35) LibreSSL/3.3.6: error:1404B42E:SSL routines:ST_CONNECT:tlsv1 alert protocol version

10. Try the same curl with http on the https listener and we see a 200 response

curl -kvv http://localhost:10443/ -H "Host:localhost2"
*   Trying [::1]:10443...
* Connected to localhost (::1) port 10443
> GET / HTTP/1.1
> Host:localhost2
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json
< date: Mon, 08 Apr 2024 21:57:21 GMT
< content-length: 131
< x-envoy-upstream-service-time: 0
< server: envoy
<
{
    "server": "chubby-kiwi-t82xhysy",
    "quote": "668: The Neighbor of the Beast.",
    "time": "2024-04-08T21:57:21.903616Z"
* Connection #0 to host localhost left intact
}%

Expected behavior
Either, the listener accepts HTTPS connections without a * host being created, or more specific routes can use mappingSelector when a * host is provided.

In Step 7, all CURLs should have been 200 response.
In step 9, we would expect a 200 response, but instead get ssl errors
In step 10, we would not expect 200 response on http when the host has a TLS-secret and we're using the HTTPS listener

Versions (please complete the following information):

• Emissary: 3.9.1, Helm version 8.9.1
• Kubernetes environment: Rancher Desktop Kubernetes 1.29.3 and EKS 1.29

Additional context
Add any other context about the problem here.

[cindymullins-dw]

We notice you're using Rancher and there's a possibility might be altering the YAML, so that's something we'd like to check. Can you run a kubectl get mapping for one of your Mappings here so we can take a look at that?

[dmaclaury]

We see this in EKS as well, but here is the requested output from my Rancher test environment:

k get mapping -A
NAMESPACE   NAME                       SOURCE HOST                      SOURCE PREFIX   DEST SERVICE   STATE   REASON
default     quote-backend-wildcard     _skip_mapping_with_empty_host_   /backend/       quote
default     quote-backend              _skip_mapping_with_empty_host_   /backend/       quote
default     quote-backend-host         _skip_mapping_with_empty_host_   /               quote
default     quote-backend-host-splat   _skip_mapping_with_empty_host_   /splat-only/    quote

k describe mapping -A
Name:         quote-backend-wildcard
Namespace:    default
Labels:       hostKind=wildcard-host
Annotations:  <none>
API Version:  getambassador.io/v2
Kind:         Mapping
Metadata:
  Creation Timestamp:  2024-04-17T20:49:45Z
  Generation:          1
  Resource Version:    152376
  UID:                 405ec9da-6446-463c-bfdb-351aa40fb27f
Spec:
  ambassador_id:
    --apiVersion-v3alpha1-only--default
  Docs:
    Path:   /.ambassador-internal/openapi-docs
  Host:     _skip_mapping_with_empty_host_
  Prefix:   /backend/
  Service:  quote
Events:     <none>


Name:         quote-backend
Namespace:    default
Labels:       hostKind=localhost2
Annotations:  <none>
API Version:  getambassador.io/v2
Kind:         Mapping
Metadata:
  Creation Timestamp:  2024-04-17T20:49:45Z
  Generation:          1
  Resource Version:    152377
  UID:                 f1e4d716-3ee7-4fdd-b983-707f0913813b
Spec:
  ambassador_id:
    --apiVersion-v3alpha1-only--default
  Docs:
    Path:   /.ambassador-internal/openapi-docs
  Host:     _skip_mapping_with_empty_host_
  Prefix:   /backend/
  Service:  quote
Events:     <none>


Name:         quote-backend-host
Namespace:    default
Labels:       hostKind=localhost2
Annotations:  <none>
API Version:  getambassador.io/v2
Kind:         Mapping
Metadata:
  Creation Timestamp:  2024-04-17T20:49:45Z
  Generation:          1
  Resource Version:    152378
  UID:                 5afd128d-ea07-4e91-860e-2a53ae367e31
Spec:
  ambassador_id:
    --apiVersion-v3alpha1-only--default
  Docs:
    Path:   /.ambassador-internal/openapi-docs
  Host:     _skip_mapping_with_empty_host_
  Prefix:   /
  Service:  quote
Events:     <none>


Name:         quote-backend-host-splat
Namespace:    default
Labels:       hostKind=localhost-splat
Annotations:  <none>
API Version:  getambassador.io/v2
Kind:         Mapping
Metadata:
  Creation Timestamp:  2024-04-17T20:49:46Z
  Generation:          1
  Resource Version:    152379
  UID:                 7feb8cc0-53dc-4930-9ad9-22151ee90e24
Spec:
  ambassador_id:
    --apiVersion-v3alpha1-only--default
  Docs:
    Path:   /.ambassador-internal/openapi-docs
  Host:     _skip_mapping_with_empty_host_
  Prefix:   /splat-only/
  Service:  quote
Events:     <none>

[cindymullins-dw]

Thanks, I think that looks ok. Can you try running this as well? kubectl get host wildcard-host -n ambassador -o yaml

[dmaclaury]

Here are all the hosts, they are in default namespace, but listeners are configured for ALL

k get hosts.getambassador.io -o yaml
apiVersion: v1
items:
- apiVersion: getambassador.io/v2
  kind: Host
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"getambassador.io/v3alpha1","kind":"Host","metadata":{"annotations":{},"name":"localhost2","namespace":"default"},"spec":{"hostname":"localhost2","mappingSelector":{"matchLabels":{"hostKind":"localhost2"}},"requestPolicy":{"insecure":{"action":"Route"}},"tlsSecret":{"name":"tls-cert"}}}
    creationTimestamp: "2024-04-17T20:49:20Z"
    generation: 1
    name: localhost2
    namespace: default
    resourceVersion: "152364"
    uid: 2d038cbe-6334-414d-928c-db845f18272a
  spec:
    ambassador_id:
    - --apiVersion-v3alpha1-only--default
    hostname: localhost2
    requestPolicy:
      insecure:
        action: Route
    selector:
      matchLabels:
        hostKind: localhost2
    tlsSecret:
      name: tls-cert
  status: {}
- apiVersion: getambassador.io/v2
  kind: Host
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"getambassador.io/v3alpha1","kind":"Host","metadata":{"annotations":{},"name":"localhost-splat","namespace":"default"},"spec":{"hostname":"*.localhost","mappingSelector":{"matchLabels":{"hostKind":"localhost-splat"}},"requestPolicy":{"insecure":{"action":"Route"}},"tlsSecret":{"name":"tls-cert"}}}
    creationTimestamp: "2024-04-17T20:49:21Z"
    generation: 1
    name: localhost-splat
    namespace: default
    resourceVersion: "152365"
    uid: 4f8ca933-1a20-43cb-baf2-22ceb7b89a6e
  spec:
    ambassador_id:
    - --apiVersion-v3alpha1-only--default
    hostname: '*.localhost'
    requestPolicy:
      insecure:
        action: Route
    selector:
      matchLabels:
        hostKind: localhost-splat
    tlsSecret:
      name: tls-cert
  status: {}
- apiVersion: getambassador.io/v2
  kind: Host
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"getambassador.io/v3alpha1","kind":"Host","metadata":{"annotations":{},"name":"wildcard-host","namespace":"default"},"spec":{"acmeProvider":{"authority":"none"},"hostname":"*","mappingSelector":{"matchLabels":{"hostKind":"wildcard-host"}},"requestPolicy":{"insecure":{"action":"Route"}},"tlsSecret":{"name":"tls-cert"}}}
    creationTimestamp: "2024-04-18T21:40:14Z"
    generation: 1
    name: wildcard-host
    namespace: default
    resourceVersion: "153787"
    uid: bf52ee0e-418d-4ef7-86f4-446f97fb8ca6
  spec:
    acmeProvider:
      authority: none
    ambassador_id:
    - --apiVersion-v3alpha1-only--default
    hostname: '*'
    requestPolicy:
      insecure:
        action: Route
    selector:
      matchLabels:
        hostKind: wildcard-host
    tlsSecret:
      name: tls-cert
  status: {}
kind: List
metadata:
  resourceVersion: ""

[cindymullins-dw]

Thanks for that. I did some research, and this seems to be a known issue: when setting mappingSelector on v3alpha1 CRDs, apiext (an Ambassador extension which converts resources to the v2 storage version) incorrectly handles the translation and stores the resource with an invalid Selector: field rather than mappingSelector. I see this in your yaml output as well.

Our recommendations for now are

1. Continue to use Selector: on the v2 resources -- Do not attempt to upgrade to v3alpha1 until the issue has been resolved with mappingSelector:
2. Manually edit the Selector: field on the v3alpha1 resource to be a mappingSelector: field. This breaks CI/CD pieplines when you introduce manual edits, and is highly discouraged as a best practice.
3. Ignore using Selector: / mappingSelector: altogether for now, and associate Mappings and Hosts by specifying Hostname: fields on the Mapping resources.