Istio Gateways Pods use too much memory, how can i fine tune them for memory usage?

[samuel-esp]

Hi everyone,

I have a big cluster (500-1000 pods range) with the Istio control plane installed. Each namespace of this Cluster can choose to adopt Istio or not, if Istio is adopted inside a namespace the CI/CD automation takes care of deploying a dedicated Gateway bounded to an Ingress Gateway Pod and a Sidecar (with all the necessary RBACs)

This is the sidecar resource

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: default
  namespace: mynamespace
spec:
  egress:
  - hosts:
    - "./*"

The Ingress Gateway template that is deployed is the same that can be generated with Istioctl install against the IstioOperator yaml containing only the Ingress Gateway set to true

Unfortunately the Sidecar resource doesn't apply to the proxy inside the Ingress Gateway as stated in the "Note" list of this doc (https://istio.io/latest/docs/reference/config/networking/sidecar/),. So, while the usage of memory for sidecar is low, I can't say the same thing for the Ingress Gateway pod. What I'm noticing is that each ingress gateway pod uses too much memory (More than 500Mi in most cases) even if the namespace contains 3/4 pods.

Can someone give me some advices to optimize the memory performance of the Ingress Gateway?

[GregHanson]

Sidecar CRD's don't apply to ingress/egress gateways because their configs are already limited by the corresponding Gateway+VirtualService which translates envoy Listeners/Route/Clusters.

What istio version are you on?

[howardjohn]

Clusters are not filtered

[samuel-esp]

Sorry I didn't specify that, I'm on Istio 1.17.

What convinced me writing this post was that today I looked inside a namespace that just started the adoption of Istio. The dev team at the moment only deployed the Gateway to balance "vanilla pods" (so no Virtual Services, Destination Rules and no Sidecars injected to pods).

The Ingress Gateway deploy of this namespace (in the 20 minutes span I looked) had 2 Replicas of the Istio Ingress Gateway where each one used 500Mi constantly. The Ingress Gateway pod is set like this:

• the Memory request and limit for the Ingress Gateway Pod was 1024Mi
• an HPA scaling mechanism based on memory that scales up another pod when more than 80% of the request is constantly used

The dev team also told me they were not doing any performance test or even basic HTTP calls today on that namespace

[samuel-esp]

I still encounter this problem almost daily, does anyone know how to optimize the ingress gateway pod?

This is the situation right now in the istio-system namespace:

• I dont have Virtual Services or Destination Rules that are linked to these Ingress pods in the istio-system namespace. I don't have a Gateway resource in the istio-system namespace
• To be more precise applicative pods don't even use or receive any type of traffic frome these ingress pods in the istio-system namespace. Each applicative pods is deployed in a dedicated namespace (that is setup with dedicated ingress gateway pods and a dedicated Gateway resource)

These ingress gateway are basically "placeholders" pod because they are present but they don't serve any purpose yet (and probably they will be removed soon). Why do we have so much memory used for each pod?

NAME                                   CPU(cores)   MEMORY(bytes)
istio-ingressgateway-d48fc9495-68hsl   30m          448Mi
istio-ingressgateway-d48fc9495-bf8lb   23m          432Mi
istio-ingressgateway-d48fc9495-sfwhs   29m          429Mi
istio-ingressgateway-d48fc9495-xlmvj   57m          750Mi
istiod-6ffc5c997-2f7ct                 1409m        2144Mi
istiod-6ffc5c997-wzpx8                 96m          409Mi

This situation is also present in each namespace, where the ingress gateway pods use much memory (in this case it could be justified by the fact that they also have Virtual Services and Destination Rules deployed).

Note: our automation (that was written before I joined) automatically links each Virtual Service created inside the namespace to the Gateway resource defined inside the namespace. Is this a bad approach? I think It could increase the usage of Memory of the Istio Ingress pods since they should store also the definitions of Virtual Service that are defined much further in the general architecture.

[samuel-esp]

Another question for dedicated gateways inside namespaces:

NAME                                     GATEWAYS                    HOSTS
1-virtualservice               ["my-namespace-ingressgateway"]   ["example.com"]
2-virtualservice               ["my-namespace-ingressgateway"]   ["example2.com"]
3-virtualservice               ["my-namespace-ingressgateway"]   ["example3.com"]
4-virtualservice               ["my-namespace-ingressgateway"]   ["example4.com"]
5-virtualservice               ["my-namespace-ingressgateway"]   ["example5.com"]    

for performance: would it be better to define a single Virtual Service with a host based routing, or multiple Virtual Services like in this case? The amount of configurations that Envoy saves is the same or not?