upstream connect error or disconnect/reset before headers. reset reason: connection termination

[rajamecs]

Hi Team,
I have 3 nodes part of EKS cluster ... From the node If I do a curl to an external request which is hosted on-prem I am getting the response on all three nodes. But when coming to PODS , only one of the pods get the response back whereas the other two pods on the other two nodes gives back this error message "upstream connect error or disconnect/reset before headers. reset reason: connection termination".

I am using
ISTIO version 1.18.1
EKS Node and Control Plan 1.24

The istio service registry has two routes for one for prometheus and other default route with domain '*' and ALLOW_ANY. The same setup in non-prod works fine on all three nodes and pods. The non-prod cluster dont have virtual service , service entry , destination rule and gateways.

After googling this error, I have introduced the Virtual Service with timeout and re-try , Service Entry , Destination Rule and Istio egress gateway but nothing helped.

Virtual Service :

apiVersion: v1
items:

• apiVersion: networking.istio.io/v1alpha3
  kind: VirtualService
  metadata:
  annotations:
  creationTimestamp: "2022-02-05T02:21:14Z"
  generation: 1
  name: nm-prod
  namespace: nm-prod
  resourceVersion: "28540"
  spec:
  hosts:
  • x.y.z.com
    http:
  • route:
    • destination:
      host: x.y.z.com
      weight: 100
      timeout: 60s
      retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: connect-failure,refused-stream,503
      kind: List
      metadata:
      resourceVersion: ""

Service Entry:

apiVersion: v1
items:

• apiVersion: networking.istio.io/v1beta1
  kind: ServiceEntry
  metadata:
  annotations:
  creationTimestamp: "2022-02-05T02:21:14Z"
  generation: 1
  name: external-se-nm-prod
  namespace: nm-prod
  resourceVersion: "28539"
  spec:
  hosts:
  • x.y.z.com
    location: MESH_EXTERNAL
    ports:
  • name: http
    number: 8080
    protocol: HTTP
    resolution: DNS
    kind: List
    metadata:
    resourceVersion: ""

Sample Request failure:

m-prod:$ curl -X POST -vvk http://x.y.z.com.com:8080/api/auth/logon?imp=xxx&clientip=0.0.0.0
[1] 480
nm-prod:$ * Trying 12.18.18.9:8080...

• Connected to x.y.z.com.com (12.18.18.9) port 8080 (#0)

POST /api/auth/logon?imp=xxxxx HTTP/1.1
Host: x.y.z.com.com:8080
User-Agent: curl/8.1.2
Accept: /

< HTTP/1.1 503 Service Unavailable
< content-length: 95
< content-type: text/plain
< date: Thu, 20 Jul 2023 18:26:32 GMT
< server: envoy
<

• Connection #0 to host x.y.z.com.com left intact
  upstream connect error or disconnect/reset before headers. reset reason: connection termination

TCP DUMP(stripped version):

10:16:27.525421 IP (tos 0x0, ttl 58, id 30777, offset 0, flags [DF], proto TCP (6), length 64)
ip-12-18-18-9.eu-central-1.compute.internal.webcache > ip-10-44-49-13.eu-central-1.compute.internal.28841: Flags [F.], cksum 0xce9e (correct), seq 1, ack 1, win 235, options [nop,nop,TS val 606764328 ecr 307198706,nop,nop,sack 1 {1349:1400}], length 0

10:16:27.525671 IP (tos 0x0, ttl 63, id 63635, offset 0, flags [DF], proto TCP (6), length 52)
ip-10-44-49-13.eu-central-1.compute.internal.28841 > ip-12-18-18-9.eu-central-1.compute.internal.webcache: Flags [F.], cksum 0x0e39 (incorrect -> 0x803a), seq 1400, ack 2, win 491, options $*}(
10:16:27.630346 IP (tos 0x0, ttl 58, id 13821, offset 0, flags [DF], proto TCP (6), length 40)
ip-12-18-18-9.eu-central-1.compute.internal.webcache > ip-10-44-49-13.eu-central-1.compute.internal.28841: Flags [R], cksum 0x2956 (correct),