[Looks like BUG] Can't meet pods in Redis Cluster using hostnames when Istio injection is enabled

[ku9nov]

Hello everyone,
I have a rather strange issue with a Redis Cluster that doesn't work when Istio injections are enabled. The minimal configuration to reproduce the problem is as follows:

apiVersion: v1
kind: Service
metadata:
  name: redis-east-redis-cluster-0-service
  namespace: redis
spec:
  selector:
    app.kubernetes.io/instance: redis-east
    app.kubernetes.io/name: redis-cluster
    statefulset.kubernetes.io/pod-name: redis-east-redis-cluster-0
  type: ClusterIP
  sessionAffinity: None
  publishNotReadyAddresses: true
  ports:
    - name: tcp-redis
      port: 6379
      targetPort: tcp-redis
      protocol: TCP
      nodePort: null
    - name: tcp-redis-bus
      protocol: TCP
      port: 16379
      targetPort: tcp-redis-bus 
---
apiVersion: v1
kind: Service
metadata:
  name: redis-east-redis-cluster-1-service
  namespace: redis
spec:
  selector:
    app.kubernetes.io/instance: redis-east
    app.kubernetes.io/name: redis-cluster
    statefulset.kubernetes.io/pod-name: redis-east-redis-cluster-1
  type: ClusterIP
  sessionAffinity: None
  publishNotReadyAddresses: true
  ports:
    - name: tcp-redis
      port: 6379
      targetPort: tcp-redis
      protocol: TCP
      nodePort: null
    - name: tcp-redis-bus
      protocol: TCP
      port: 16379
      targetPort: tcp-redis-bus 
---
apiVersion: v1
kind: Service
metadata:
  name: redis-east-redis-cluster-2-service
  namespace: redis
spec:
  selector:
    app.kubernetes.io/instance: redis-east
    app.kubernetes.io/name: redis-cluster
    statefulset.kubernetes.io/pod-name: redis-east-redis-cluster-2
  type: ClusterIP
  sessionAffinity: None
  publishNotReadyAddresses: true
  ports:
    - name: tcp-redis
      port: 6379
      targetPort: tcp-redis
      protocol: TCP
      nodePort: null
    - name: tcp-redis-bus
      protocol: TCP
      port: 16379
      targetPort: tcp-redis-bus 
---
apiVersion: v1
kind: Service
metadata:
  name: redis-east-redis-cluster-3-service
  namespace: redis
spec:
  selector:
    app.kubernetes.io/instance: redis-east
    app.kubernetes.io/name: redis-cluster
    statefulset.kubernetes.io/pod-name: redis-east-redis-cluster-3
  type: ClusterIP
  sessionAffinity: None
  publishNotReadyAddresses: true
  ports:
    - name: tcp-redis
      port: 6379
      targetPort: tcp-redis
      protocol: TCP
      nodePort: null
    - name: tcp-redis-bus
      protocol: TCP
      port: 16379
      targetPort: tcp-redis-bus 

---
apiVersion: v1
kind: Service
metadata:
  name: redis-east-redis-cluster-4-service
  namespace: redis
spec:
  selector:
    app.kubernetes.io/instance: redis-east
    app.kubernetes.io/name: redis-cluster
    statefulset.kubernetes.io/pod-name: redis-east-redis-cluster-4
  type: ClusterIP
  sessionAffinity: None
  publishNotReadyAddresses: true
  ports:
    - name: tcp-redis
      port: 6379
      targetPort: tcp-redis
      protocol: TCP
      nodePort: null
    - name: tcp-redis-bus
      protocol: TCP
      port: 16379
      targetPort: tcp-redis-bus 

---
apiVersion: v1
kind: Service
metadata:
  name: redis-east-redis-cluster-5-service
  namespace: redis
spec:
  selector:
    app.kubernetes.io/instance: redis-east
    app.kubernetes.io/name: redis-cluster
    statefulset.kubernetes.io/pod-name: redis-east-redis-cluster-5
  type: ClusterIP
  sessionAffinity: None
  publishNotReadyAddresses: true
  ports:
    - name: tcp-redis
      port: 6379
      targetPort: tcp-redis
      protocol: TCP
      nodePort: null
    - name: tcp-redis-bus
      protocol: TCP
      port: 16379
      targetPort: tcp-redis-bus 
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-east-redis-cluster
  namespace: redis
  labels:
    app.kubernetes.io/instance: redis-east
    app.kubernetes.io/name: redis-cluster
    app.kubernetes.io/version: 7.2.4

spec:
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/instance: redis-east
      app.kubernetes.io/name: redis-cluster
  replicas: 6
  serviceName: redis-east-redis-cluster-headless
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: redis-east
        app.kubernetes.io/name: redis-cluster
        app.kubernetes.io/version: 7.2.4
    spec:
      hostNetwork: false
      containers:
      - name: redis
        image: redis:7.2.4
        imagePullPolicy: "IfNotPresent"
        command: ['/bin/bash', '-c']
        args:
          - |
            cp /conf/redis-default.conf /etc/redis.conf
            export REDIS_CLUSTER_ANNOUNCE_HOSTNAME="$POD_NAME-service.redis"
            pod_index=($(echo "$POD_NAME" | tr "-" "\n"))
            pod_index="${pod_index[-1]}"
            if [[ "$pod_index" == "0" ]]; then
              export REDIS_CLUSTER_CREATOR="yes"
              export REDIS_CLUSTER_REPLICAS="1"
            fi
            echo -en "\n" >> /etc/redis.conf
            echo "cluster-announce-hostname $REDIS_CLUSTER_ANNOUNCE_HOSTNAME" >> /etc/redis.conf
            redis-server /etc/redis.conf
        ports:
        - containerPort: 6379
          name: tcp-redis
        - containerPort: 16379
          name: tcp-redis-bus
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        volumeMounts:
        - name: conf
          mountPath: /conf
          readOnly: false
      volumes:
      - name: conf
        configMap:
          name: redis-east-redis-cluster-default
          defaultMode: 0755

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: redis-east-redis-cluster-default
  namespace: redis
  labels:
    app.kubernetes.io/instance: redis-east
    
    app.kubernetes.io/name: redis-cluster
    app.kubernetes.io/version: 7.2.4
    
data:
  redis-default.conf: |-
    bind 0.0.0.0 ::
    port 6379
    cluster-enabled yes
    cluster-config-file nodes.conf
    cluster-node-timeout 5000
    requirepass strong-password
    appendonly yes
    cluster-preferred-endpoint-type hostname

Actual behaviour(Stucked in join state):

kubectl exec -it redis-east-redis-cluster-0  bash -n redis
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@redis-east-redis-cluster-0:/data# redis-cli --cluster create redis-east-redis-cluster-0-service.redis:6379 redis-east-redis-cluster-1-service.redis:6379 redis-east-redis-cluster-2-service.redis:6379 redis-east-redis-cluster-3-service.redis:6379 redis-east-redis-cluster-4-service.redis:6379 redis-east-redis-cluster-5-service.redis:6379 --cluster-replicas 1 -a strong-password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica redis-east-redis-cluster-4-service.redis:6379 to redis-east-redis-cluster-0-service.redis:6379
Adding replica redis-east-redis-cluster-5-service.redis:6379 to redis-east-redis-cluster-1-service.redis:6379
Adding replica redis-east-redis-cluster-3-service.redis:6379 to redis-east-redis-cluster-2-service.redis:6379
M: 4eb5065a23c23b57319badd5a78b5e82070fe069 redis-east-redis-cluster-0-service.redis:6379
   slots:[0-5460] (5461 slots) master
M: d416a11950f62ad7c613d5357f47fc839c9d2ff9 redis-east-redis-cluster-1-service.redis:6379
   slots:[5461-10922] (5462 slots) master
M: 087a76bb27c1c1e60662c24e4848a3a9f7ef81e4 redis-east-redis-cluster-2-service.redis:6379
   slots:[10923-16383] (5461 slots) master
S: d50707882554213220f63b728d3a458565139595 redis-east-redis-cluster-3-service.redis:6379
   replicates 087a76bb27c1c1e60662c24e4848a3a9f7ef81e4
S: 72f3daae2f32d054d701bbcde85c43f661e4803e redis-east-redis-cluster-4-service.redis:6379
   replicates 4eb5065a23c23b57319badd5a78b5e82070fe069
S: 49f009779b71cd1944115794dc5f6a57e4ef2dcd redis-east-redis-cluster-5-service.redis:6379
   replicates d416a11950f62ad7c613d5357f47fc839c9d2ff9
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.................^C

Expected behaviour(working when Injection is disabled):

kubectl exec -it redis-east-redis-cluster-0  bash -n redis
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@redis-east-redis-cluster-0:/data# redis-cli --cluster create redis-east-redis-cluster-0-service.redis:6379 redis-east-redis-cluster-1-service.redis:6379 redis-east-redis-cluster-2-service.redis:6379 redis-east-redis-cluster-3-service.redis:6379 redis-east-redis-cluster-4-service.redis:6379 redis-east-redis-cluster-5-service.redis:6379 --cluster-replicas 1 -a strong-password
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica redis-east-redis-cluster-4-service.redis:6379 to redis-east-redis-cluster-0-service.redis:6379
Adding replica redis-east-redis-cluster-5-service.redis:6379 to redis-east-redis-cluster-1-service.redis:6379
Adding replica redis-east-redis-cluster-3-service.redis:6379 to redis-east-redis-cluster-2-service.redis:6379
M: 4d57f781477b9dc50fec8c8a1444adf1ac6bb972 redis-east-redis-cluster-0-service.redis:6379
   slots:[0-5460] (5461 slots) master
M: 89c267084ce0f620508c67ee514726c45e36710d redis-east-redis-cluster-1-service.redis:6379
   slots:[5461-10922] (5462 slots) master
M: 64d715c828ada458ff03c9c03ba9752f376884f9 redis-east-redis-cluster-2-service.redis:6379
   slots:[10923-16383] (5461 slots) master
S: 3136b9dcd2e06fcddec0d97550df7a4806cd4d95 redis-east-redis-cluster-3-service.redis:6379
   replicates 64d715c828ada458ff03c9c03ba9752f376884f9
S: d956cd753f4c32d26be2c36ba68db22796960e16 redis-east-redis-cluster-4-service.redis:6379
   replicates 4d57f781477b9dc50fec8c8a1444adf1ac6bb972
S: b85c34b0d9fbc068fbb2702dcafec243b61df5df redis-east-redis-cluster-5-service.redis:6379
   replicates 89c267084ce0f620508c67ee514726c45e36710d
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join

>>> Performing Cluster Check (using node redis-east-redis-cluster-0-service.redis:6379)
M: 4d57f781477b9dc50fec8c8a1444adf1ac6bb972 redis-east-redis-cluster-0-service.redis:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
M: 64d715c828ada458ff03c9c03ba9752f376884f9 10.244.1.180:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: b85c34b0d9fbc068fbb2702dcafec243b61df5df 10.244.0.9:6379
   slots: (0 slots) slave
   replicates 89c267084ce0f620508c67ee514726c45e36710d
M: 89c267084ce0f620508c67ee514726c45e36710d 10.244.0.38:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: d956cd753f4c32d26be2c36ba68db22796960e16 10.244.0.5:6379
   slots: (0 slots) slave
   replicates 4d57f781477b9dc50fec8c8a1444adf1ac6bb972
S: 3136b9dcd2e06fcddec0d97550df7a4806cd4d95 10.244.0.123:6379
   slots: (0 slots) slave
   replicates 64d715c828ada458ff03c9c03ba9752f376884f9
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

But i can connect to all pods with redis-cli -h service-name.namespace which use 6379 port.

Interestingly, if, for example, I configure the Redis cluster to use IP addresses, it works. However, when I try to connect them using service names, I can connect to the pods by service name, but Redis cannot connect to other pods using the service name and port 16379.

If I disable Istio injection, it works fine. However, I need injection and service names to support multiple regions.

[ku9nov]

1. Disabling MTLS doesn't help.

2. Excluding port 16379 from the Istio proxy which can be achieved with adding it to StatefulSet:

annotations:
  traffic.sidecar.istio.io/excludeOutboundPorts: "16379" 
  traffic.sidecar.istio.io/excludeInboundPorts: "16379"

This solution is effective for a single cluster. However, in a multicluster deployment, it doesn't work, of course.
Any ideas?

[ku9nov]

Redis Sentinel could work and communicate with pods using hostnames. It seems like there is a bug specifically related to the Redis Cluster with Istio installation.