Cannot get Istio Gateway/VirtualService to work with a Let's encrypt certificate #50563

croessner · 2024-04-19T13:46:29Z

croessner
Apr 19, 2024

Hi,

I am relatively new to Kubernetes and Istio. About a month ago I installed an on-prem Cluster with 3 masters and 3 workers. Then I installed the cert-manager from Helm. I have a ClusterIssuer for prod and staging.

I have done a deployment that works with Istio and I can connect from my remote HAproxy LBR to the Istio.

backend be_nauthilus_prod from defaults_nauthilus
  option httpchk
  # http-check connect ssl alpn h2,http/1.1
  http-check send meth GET uri /ping ver HTTP/1.1 hdr Host login.authserv.me body "pong"
  http-check expect status 200
  cookie server_used dynamic indirect nocache insert
  dynamic-cookie-key *******
  # server kn1.roessner-net.de 192.168.0.180:31246 check inter 1m ssl alpn h2 verify none rise 2 fall 2 on-marked-down shutdown-sessions
  # server kn2.roessner-net.de 192.168.0.181:31246 check inter 1m ssl alpn h2 verify none rise 2 fall 2 on-marked-down shutdown-sessions
  # server kn3.roessner-net.de 192.168.0.182:31246 check inter 1m ssl alpn h2 verify none rise 2 fall 2 on-marked-down shutdown-sessions
  server kn1.roessner-net.de 192.168.0.180:30264 check inter 1m rise 2 fall 2 on-marked-down shutdown-sessions
  server kn2.roessner-net.de 192.168.0.181:30264 check inter 1m rise 2 fall 2 on-marked-down shutdown-sessions
  server kn3.roessner-net.de 192.168.0.182:30264 check inter 1m rise 2 fall 2 on-marked-down shutdown-sessions

Istio was installed like this (Version 1.21.1):

istioctl install  \
	--set meshConfig.accessLogFile=/dev/stdout \
	--set values.gateways.istio-ingressgateway.type=NodePort

Here is a part of my deployment-manifest:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nauthilus
  namespace: auth
spec:
  selector:
    matchLabels:
      app: nauthilus
  template:
    metadata:
      labels:
        app: nauthilus
...
          ports:
            - containerPort: 9080
...
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: login-authserv-tls
  namespace: auth
spec:
  dnsNames:
  - login.authserv.me
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-staging
  secretName: login-authserv-tls
---
apiVersion: v1
kind: Service
metadata:
  name: nauthilus-service
  namespace: auth
spec:
  selector:
    app: nauthilus
  ports:
    - name: nt-http
      port: 9080
      targetPort: 9080
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: nauthilus-gateway
  namespace: auth
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - login.authserv.me
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - login.authserv.me
      tls:
        mode: SIMPLE
        credentialName: login-authserv-tls
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nauthilus-virtual-service
  namespace: auth
spec:
  hosts:
    - login.authserv.me
  gateways:
    - nauthilus-gateway
  http:
    - match:
      - headers:
          Host:
            exact: login.authserv.me
      route:
        - destination:
            host: nauthilus-service
            port:
              number: 9080
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: nauthilus-permissive
  namespace: auth
spec:
  selector:
    matchLabels:
      app: nauthilus
  mtls:
    mode: PERMISSIVE  # Currently there doe not exist any default PeerAuthentication in the cluster!

The certificate was created:

kubectl -n auth get certificate                                                                                                     15:24:34
NAME                 READY   SECRET               AGE
login-authserv-tls   True    login-authserv-tls   6h49m

Listeners call:

istioctl pc listeners -n istio-system  istio-ingressgateway-d7cd98789-nbgqj
ADDRESSES PORT  MATCH                  DESTINATION
0.0.0.0   8080  ALL                    Route: http.8080
0.0.0.0   8443  SNI: login.authserv.me Route: https.443.https.nauthilus-gateway.auth
0.0.0.0   15021 ALL                    Inline Route: /healthz/ready*
0.0.0.0   15090 ALL                    Inline Route: /stats/prometheus*

Secrets:

istioctl pc secret -n istio-system istio-ingressgateway-d7cd98789-nbgqj
zsh: correct 'secret' to 'secrets' [nyae]? n
RESOURCE NAME                       TYPE           STATUS     VALID CERT     SERIAL NUMBER                            NOT AFTER                NOT BEFORE
default                             Cert Chain     ACTIVE     true           ae46e57af984b867501b383411056ea0         2024-04-20T13:01:57Z     2024-04-19T12:59:57Z
kubernetes://login-authserv-tls     Cert Chain     ACTIVE     true           2b7caef2b0c086605e60b148463ed3554ebe     2024-07-18T06:51:24Z     2024-04-19T06:51:25Z
ROOTCA                              CA             ACTIVE     true           8e5399a24d2d28f618e0771fd4c8abfb         2034-04-16T13:01:48Z     2024-04-18T13:01:48Z

Pods in istio-system:

kubectl get pods -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
grafana-6f68dfd8f4-l7mw7               1/1     Running   0          21h
istio-ingressgateway-d7cd98789-nbgqj   1/1     Running   0          24h
istiod-68cb48856d-8vmqm                1/1     Running   0          24h
jaeger-7d7d59b9d-q9htd                 1/1     Running   0          21h
kiali-588bc98cd-6hljc                  1/1     Running   0          21h
prometheus-7545dd48db-hz7cv            2/2     Running   0          21h

Pods in auth:

kubectl get pods -n auth
NAME              READY   STATUS    RESTARTS        AGE
nauthilus-dlwcd   2/2     Running   3 (6h59m ago)   22h
nauthilus-f7fng   2/2     Running   1 (10h ago)     22h
nauthilus-vw5km   2/2     Running   1 (34m ago)     22h

When trying to connect to the service using HTTPS, I see the follwing errors in my istio-gateway logs:

kubectl logs istio-ingressgateway-d7cd98789-nbgqj -n istio-system | grep filter_chain_not_found | tail
[2024-04-19T10:13:22.645Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 9 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:1394 - -
[2024-04-19T10:13:23.683Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 6 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:28265 - -
[2024-04-19T10:13:24.708Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 3 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:53749 - -
[2024-04-19T10:13:25.737Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 6 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:35022 - -
[2024-04-19T10:13:26.761Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 3 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:50896 - -
[2024-04-19T10:13:27.784Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 5 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:63985 - -
[2024-04-19T10:13:28.807Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 4 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:39671 - -
[2024-04-19T10:13:29.828Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 3 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:21819 - -
[2024-04-19T10:13:30.852Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 3 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:63980 - -
[2024-04-19T10:13:31.879Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 7 - "-" "-" "-" "-" "-" - - 172.17.60.240:8443 192.168.122.180:64593 - -

Node-Ports for the istio-ingressgateway:

kubectl -n istio-system get svc | grep istio-ingressgateway
istio-ingressgateway   NodePort    10.107.222.131   <none>        15021:31267/TCP,80:30264/TCP,443:31246/TCP       24h

Testing from the LBR:

curl -v -H "Host: login.authserv.me" -k https://192.168.0.180:31246/ping
*   Trying 192.168.0.180:31246...
* Connected to 192.168.0.180 (192.168.0.180) port 31246 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to 192.168.0.180:31246
* Closing connection 0
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 192.168.0.180:31246

Whatever I tried, I can not get HTTPS to work.

All examples I found on the internet most often use Kubernetes in a KaaS with external LBR. So maybe I miss something for the NodePort HAproxy-setup for on-prem-systems.

Kubernetes nodes information:

kubectl get nodes
NAME                  STATUS   ROLES           AGE   VERSION
km1.roessner-net.de   Ready    control-plane   53d   v1.29.3
km2.roessner-net.de   Ready    control-plane   53d   v1.29.3
km3.roessner-net.de   Ready    control-plane   38d   v1.29.3
kn1.roessner-net.de   Ready    loadbalancer    53d   v1.29.3
kn2.roessner-net.de   Ready    loadbalancer    53d   v1.29.3
kn3.roessner-net.de   Ready    loadbalancer    53d   v1.29.3

If someone does have an idea, what is wrong, I really thank you in advance! If someone thinks, it might be a bug, I will report this question as a bug.

Answered by howardjohn

Apr 19, 2024

This is a case of https://istio.io/latest/docs/ops/common-problems/network-issues/#configuring-sni-routing-when-not-sending-sni. Note -H on curl only sets the HTTP host header, not the TLS SNI header

View full answer

howardjohn · 2024-04-19T14:02:17Z

howardjohn
Apr 19, 2024
Maintainer

This is a case of https://istio.io/latest/docs/ops/common-problems/network-issues/#configuring-sni-routing-when-not-sending-sni. Note -H on curl only sets the HTTP host header, not the TLS SNI header

1 reply

croessner Apr 19, 2024
Author

You really saved my life :-) Thank you so much for pointing that out. In fact my setup is completely right. I only needed to change the curl-call and for HAproxy itself, I entirely removed the httpchk and modified the server-lines to send the correct SNI. Everything works now as expected.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot get Istio Gateway/VirtualService to work with a Let's encrypt certificate #50563

{{title}}

Replies: 1 comment 1 reply

{{title}}

{{title}}

Select a reply

Cannot get Istio Gateway/VirtualService to work with a Let's encrypt certificate #50563

croessner Apr 19, 2024

Replies: 1 comment · 1 reply

howardjohn Apr 19, 2024 Maintainer

croessner Apr 19, 2024 Author

croessner
Apr 19, 2024

Replies: 1 comment 1 reply

howardjohn
Apr 19, 2024
Maintainer

croessner Apr 19, 2024
Author