Circuit breaker not working

[noor-muradi]

I have deployed a sample app (helloword) with two versions (v1 and v2), I have defined circuit breaker to stop sending traffic to unhealthy deployment, after I delete manually the v2 deployment, and observing the kiali graph, it does not seem to stop the traffic to unhealthy deployment (v2).

Deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    version: v1
  name: hello-v1
  namespace: practice
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
        version: v1
    spec:
      containers:
      - image: noormuradi/v1
        name: hello-v1

---
apiVersion: v1
kind: Service
metadata:
  name: helloworld
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: helloworld

---

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    version: v2
  name: hello-v2
  namespace: practice
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
  template:
    metadata:
      labels:
        app: helloworld
        version: v2
    spec:
      containers:
      - image: noormuradi/v2
        name: hello-v2

Gateway-vs-dr.yaml:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: hello-gateway
  namespace: practice
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
      - "hello.example.com"

---

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
 name: hello
 namespace: practice
spec:
 hosts:
 - "hello.example.com"
 gateways:
 - hello-gateway
 http:
 - route:
   - destination:
       host: helloworld
       subset: v1
     weight: 80
   - destination:
       host: helloworld
       subset: v2
     weight: 20
---

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: hello
  namespace: practice
spec:
  host: helloworld
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100

while true; do curl hello.example.com; sleep 1; done
<h1>V1</h1>
<h1>V1</h1>
<h1>V1</h1>
<h1>V1</h1>
<h1>V1</h1>
no healthy upstream
<h1>V1</h1>
<h1>V1</h1>
<h1>V1</h1>
<h1>V1</h1>
<h1>V1</h1>
no healthy upstream
<h1>V1</h1>
<h1>V1</h1>
<h1>V1</h1>
<h1>V1</h1>
no healthy upstream
<h1>V1</h1>
<h1>V1</h1>

[howardjohn]

The circuit breaker is about evicting endpoints within a group (service+subset+port).

So you have a group (helloworldv2), with 0 endpoints. You explicitly send to this (via the VS), and there is no endpoints so it fails. The circuit breaker is irrelevant here.

You can kind of work around this in a way using "locality lb". This was originally intended for other purposes but works here:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: echo-versions
spec:
  host: echo-versions.default.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http:
        maxRequestsPerConnection: 1
    loadBalancer:
      simple: ROUND_ROBIN
      localityLbSetting:
        enabled: true
        failoverPriority:
        - version=v1
        - version=v2
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 1s
      baseEjectionTime: 1m

And then do not use subsets at all. This will send requests first to things matching version=v1, then to version=v2 if those are failing.

[noor-muradi]

@howardjohn With implementing "locality lb", I will lose the weighted traffic splitting and control, so it will basically use the active passiv (failover) traffic distribution?

[howardjohn]

No, traffic splitting picks the group, not the endpoint within the group. so you could use both together if you wanted

[igordayen]

Hello John.
I applied configuration per recommendation for Bookinfo review service:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"networking.istio.io/v1beta1","kind":"DestinationRule","metadata":{"annotations":{},"creationTimestamp":"2024-04-20T00:20:45Z","generation":1,"managedFields":[{"apiVersion":"networking.istio.io/v1alpha3","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:host":{},"f:subsets":{}}},"manager":"kubectl-client-side-apply","operation":"Update","time":"2024-04-20T00:20:45Z"}],"name":"reviews","namespace":"bookinfo","resourceVersion":"55546","uid":"af9cee21-4bdd-462d-b89f-eedcf917f4ee"},"spec":{"host":"reviews","subsets":[{"labels":{"version":"v1"},"name":"v1"},{"labels":{"version":"v2"},"name":"v2"},{"labels":{"version":"v3"},"name":"v3"}],"trafficPolicy":{"connectionPool":{"http":{"http1MaxPendingRequests":1,"maxRequestsPerConnection":1},"tcp":{"maxConnections":1}},"outlierDetection":{"baseEjectionTime":"3m","consecutive5xxErrors":1,"interval":"1s","maxEjectionPercent":100}}}}
  creationTimestamp: '2024-04-20T00:20:45Z'
  generation: 2
  managedFields:
    - apiVersion: networking.istio.io/v1alpha3
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations': {}
        'f:spec':
          .: {}
          'f:host': {}
          'f:subsets': {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2024-04-20T00:20:45Z'
    - apiVersion: networking.istio.io/v1beta1
      fieldsType: FieldsV1
      fieldsV1:
        'f:metadata':
          'f:annotations':
            'f:kubectl.kubernetes.io/last-applied-configuration': {}
        'f:spec':
          'f:trafficPolicy':
            .: {}
            'f:connectionPool':
              .: {}
              'f:http':
                .: {}
                'f:http1MaxPendingRequests': {}
                'f:maxRequestsPerConnection': {}
              'f:tcp':
                .: {}
                'f:maxConnections': {}
            'f:outlierDetection':
              .: {}
              'f:baseEjectionTime': {}
              'f:consecutive5xxErrors': {}
              'f:interval': {}
              'f:maxEjectionPercent': {}
      manager: kubectl-client-side-apply
      operation: Update
      time: '2024-04-20T22:18:40Z'
  name: reviews
  namespace: bookinfo
  resourceVersion: '95768'
  uid: af9cee21-4bdd-462d-b89f-eedcf917f4ee
spec:
  host: reviews
  trafficPolicy:
    connectionPool:
      http:
        maxRequestsPerConnection: 1
    loadBalancer:
      simple: ROUND_ROBIN
      localityLbSetting:
        enabled: true
        failoverPriority:
        - version=v1
        - version=v2
        - version=v3
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 1s
      baseEjectionTime: 2m
      maxEjectionPercent: 100

however, did get expected results, all passed, using Fortio:
IP addresses distribution:
192.168.130.11:80: 10
Code 200 : 40 (100.0 %)
Response Header Sizes : count 40 avg 271.95 +/- 0.2179 min 271 max 272 sum 10878
Response Body/Total Sizes : count 40 avg 5561.95 +/- 0.2179 min 5561 max 5562 sum 222478
All done 40 calls (plus 0 warmup) 200.177 ms avg, 38.8 qps

Appreciate if you may advise. Thank you

[igordayen]

however, did NOT get expected results

[igordayen]

Environment details:
Environment: OpenShift Local CRC version: 2.31.0+6d23b6 OpenShift version: 4.14.7
Podman version: 4.4.4
Red Hat OpenShift Service Mesh 2.5.0-0 provided by Red Hat, Inc.