Reduce Control Plane Requests

[vlerenc]

What do you suggest to change:

• KAPI
  • Reduce KAPI HVPA minAllowed for memory (there isn't one for CPU anyway) from 400M down to 200M: Reduce minAllowed values from kube-apiserver and kube-controller-manager #9654
  • Change KAPI HVPA maximum number of replicas and set them from 4 to 3 to better match HA clusters in 3 zones (we believe we do not need 4, if under VPA anyway, so let's get rid of that odd number): Set kubeApiserver maxReplicas: 3 #9605
  • Quick-fix vertical scaling after KAPIs reach 4 (3 if the above gets implemented) replicas and then going below (can then only happen for non-HA clusters), effectively freezing the vertical scaling (wasting resources from then onwards).
  • Replace HVPA with HPA and VPA on CPU and metrics (both on both):
    • Dropped: Temporary solution: https://sap-ti.slack.com/archives/C017KSLTF4H/p1713531512758829
    • Accepted: Actual solution: https://gist.github.tools.sap/D043832/2051e639cf77d878f188234125c915f6
    • On Hold: Amending solution: GEP-23 tracked here
• KCM:
  • Reduce KCM HVPA minAllowed for memory (there isn't one for CPU anyway) from 100M down to 50M: Reduce minAllowed values from kube-apiserver and kube-controller-manager #9654

Why is this needed:
Primarily KAPIs but also other components waste too many resources. It would be great to address these problems, especially for KAPIs until GEP-23 lands.

[ialidzhikov]

/assign