AWS External Load Balancer TLS Termination - generate 443 server rules when default-external-scheme: https is set

[patrickleet]

What version of Knative?

0.13.x

Expected Behavior

Hey all, I have istio installed on my eks cluster, and aws lb controller, to set up the loadbalancer including an annotation to set a tls cert from ACM to the lb. I can create virtual services and gateways that terminate traffic at the LB, but having trouble trying to configure knative serving to work this way as well.

I tried setting default-external-scheme: https, which makes the urls https, but I'm getting 404s for all of them.

I figured out the gateway that is created by knative using this setup only contains port 80, I had to edit it to allow port 443
here's what's generated:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  creationTimestamp: "2024-04-10T20:27:47Z"
  generation: 6
  labels:
    networking.internal.knative.dev/ingress: example-springboot-dev
  name: example-springboot-dev-3797421420
  namespace: vm-dev-env
  ownerReferences:
  - apiVersion: networking.internal.knative.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Ingress
    name: example-springboot-dev
    uid: 1aa7bdb4-3f58-4907-a03d-c642b9590488
  resourceVersion: "18387413"
  uid: e7245ddb-541e-4ab9-9fed-0cde7e98f7cf
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
  - hosts:
    - example-springboot-dev.example.com
    port:
      name: http-server
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true

To get it working, I needed to add:

  - hosts:
    - example-springboot-dev.example.com
    port:
      name: https-server
      number: 443
      protocol: HTTP

I would expect that the 443 rule would be added to the gateway when default-external-scheme: https is set, or another option would exist to specify that the rule should be added.

Actual Behavior

See Expected Behavior as that covers what I expect as well as what actually happens, and the manual workaround required to get things moving - summed up, I can't find a way to add the 443 rule without AutoTLS, but that isn't what I want.

Steps to Reproduce the Problem

Set up KNative Serving using default-external-scheme: https and httpProtocol: Redirected

Additional Information

There is a thread in slack here: https://cloud-native.slack.com/archives/C04LMU0AX60/p1712785405535979

@dprotaso suggested making another loadbalancer as a workaround, but I 1) don't know how to do this as it's all through AWS LB controller and annotations passed to istio ingress gateway helm chart, 2) I don't really want an additional load balancer as the existing one would work if the 443 server rules were generated. In summary, it feels like just generating the 443 rules I add manually to make things work should be supported.

[skonto]

I would expect that the 443 rule would be added to the gateway when default-external-scheme: https is set, or another option would exist to specify that the rule should be added.

cc @ReToCode may have to add more.

[ReToCode]

I don't think this is a from us supported scenario.

Knative itself can provide TLS on external domains in it's ingress layer. Assuming istio you can put a TLS certificate on istio using this guide: https://knative.dev/docs/serving/encryption/external-domain-tls/#obtaining-a-certificate-using-a-tool (check the istio tab). If now your LB does another TLS termination, you have to make sure that the LB connects using https to the TLS cert that you put in the istio gateway and that your AWS LB trusts that certificate somehow.