Mutual TLS TroubleShooting

Mutual TLS troubleshooting

Auto mTLS

If you suspect auto mTLS is not working as expected, please first read the documentation.

Auto mTLS has a known limitation with workload level peer authentication. When workload level peer authentication is used, you need to configure a corresponding destination rule configuration to let client sidecar send mTLS or plaintext traffic correctly.

If you still think there's an issue, follow the instructions below for investigation.

1. Get the client envoy cluster configuration to ensure auto mTLS is configured. You should see transportSocketMatches configured for the given Envoy cluster.

   $ istioctl proxy-config clusters ${POD}.foo -ojson | grep 'name.*outbound.*httpbin' -A130
   "name": "outbound|8000||httpbin.default.svc.cluster.local",
   ...
   "transportSocketMatches": [
           {
               "name": "tlsMode-istio",
               "match": {
                   "tlsMode": "istio"
               },
               "transportSocket": {
                   "name": "envoy.transport_sockets.tls",
                   "typedConfig": {
                       "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
                       "commonTlsContext": ...
           }
           {
               "name": "tlsMode-disabled",
               "match": {},
               "transportSocket": {
                   "name": "envoy.transport_sockets.raw_buffer"
               }
           }

2. Check the destination pod to see if the destination endpoint has the label tlsMode equals to istio. For example, in the EDS response received by sleep pod, we see httpbin pod endpoint has the tlsMode label.

   $ IP=$(kubectl get pod -lapp=sleep -o jsonpath="{.items[*].status.podIP}")
   $ kubectl exec -it ${POD} -c istio-proxy -- curl 'localhost:15000/config_dump?include_eds=true' | grep ${IP} -A15 -B5
     "endpoint": {
       "address": {
       "socket_address": {
           "address": "10.32.28.136",
           "port_value": 80
       }
       ...
       "metadata": {
         "filter_metadata": {
           "envoy.transport_socket_match": {
             "tlsMode": "istio"
           }
       },

3. Check Envoy upstream debug log to confirm what transport socket configuration is being used. This requires to change Envoy upstream log to debug level.

   $ istioctl pc log ${POD} --level debug
   $ kubectl logs -f ${POD} -c istio-proxy
   2021-04-08T22:35:22.650478Z     debug   envoy upstream  transport socket match, socket tlsMode-istio selected for host with address 10.32.28.136:80

   The log line above indicates that mTLS socket is selected for the connection to the corresponding host. Please note that the connection creation might be done in advance of the actual request, if you don't see such log line, try to recreate destination side endpoint.