Extremely Slow response through emissary-ingress (+5 seconds)

[darkn3rd]

Describe the bug

The response through emissary-ingress is very slow adding several seconds (consistently an extra 5 seconds) to each connection.

To Reproduce
Steps to reproduce the behavior:

1. Install application that uses HTTP, for example, Dgraph on GKE:

   helm repo add dgraph https://charts.dgraph.io && helm update
   
   # Install Dgraph
   helm install dg dgraph/dgraph \
     --namespace dgraph \
     --create-namespace \
     --values -  <<EOF
   zero:
     persistence:
       storageClass: premium-rwo
       size: 10Gi
   alpha:
     configFile:
       config.yaml: |
         security:
           whitelist: "0.0.0.0/0"
     persistence:
       storageClass: premium-rwo
       size: 30Gi
   EOF

2. Add appropriate CRDs for Dgraph

   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Host
   metadata:
     name: dg-dgraph
   spec:
     hostname: "*"
     requestPolicy:
       insecure:
         action: Route
   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Listener
   metadata:
     name: dg-dgraph
   spec:
     port: 8080
     protocol: HTTP
     securityModel: INSECURE
     hostBinding:
       namespace:
         from: SELF
   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Mapping
   metadata:
     name: dg-dgraph-http
   spec:
     hostname: dgraph.local
     prefix: /
     service: dg-dgraph-alpha.dgraph:8080
   ---
   apiVersion: getambassador.io/v3alpha1
   kind: Mapping
   metadata:
     name: dg-dgraph-grpc
   spec:
     hostname: "*"
     prefix: /api.Dgraph/
     rewrite: /api.Dgraph/
     service: dg-dgraph-alpha.dgraph:9080
     grpc: True

3. Either update DNS record or /etc/hosts with emissary-ingress LB address

   export SERVICE_IP=$(kubectl get svc emissary-ingress \
     --namespace emissary \
     --output jsonpath='{.status.loadBalancer.ingress[0].ip}'
   )
   
   sudo sh -c "echo ${SERVICE_IP} dgraph.local" >> /etc/hosts"

4. Attempt to reach the service through emissary-ingress

   DGRAPH_HOSTNAME_HTTP="dgraph.local"
   time curl -s http://$DGRAPH_HOSTNAME_HTTP/state | jq

Expected behavior

I expected that the time not add 5 seconds to the response. I am getting this from time:

real	0m5.327s
user	0m0.034s
sys	0m0.020s

Just to demonstrate that it is not the service itself, I can exec to a pod within the Dgraph cluster, and curl the service proxy and get a fast response:

# exec into container
kubectl exec -ti dg-dgraph-zero-0 --namespace dgraph -- bash

# inside container
export DGRAPH_HOSTNAME_HTTP=dg-dgraph-alpha-headless.dgraph.svc:8080
time curl -s http://$DGRAPH_HOSTNAME_HTTP/state | jq

From this, you can see the time result is:

real	0m0.066s
user	0m0.060s
sys	0m0.008s

Versions (please complete the following information):

• Ambassador: 3.9.1
• Kubernetes environment GKE v1.27.3-gke.100

Additional context

• GKE was provisioned using this guide: https://joachim8675309.medium.com/ultimate-baseline-gke-cluster-261c1b5544be
• Emissary-Ingress was installed using helm: https://www.getambassador.io/docs/emissary/latest/tutorials/getting-started
• Dgraph was deployed using the following below, but any web service should illustrate the problem.

[darkn3rd]

Anyone around to try this out?