Introduce a new autoscaling mode (VPAAndHPA) for Shoot Kubernetes API servers
#9678
Conversation
|
Skipping CI for Draft Pull Request. |
gardener-prow
bot
added
do-not-merge/work-in-progress
gardener-prow
bot
added
the
size/XL
|
/assign |
gardener-prow
bot
added
cla: no
|
/assign |
ialidzhikov
force-pushed
the
enh/hvpa-alternative-for-apiserver
branch
from
e721c46
to
ad38c30
Compare
gardener-prow
bot
removed
the
needs-rebase
ialidzhikov
force-pushed
the
enh/hvpa-alternative-for-apiserver
branch
2 times, most recently
from
e3732e1
to
1b8d6c8
Compare
ialidzhikov
force-pushed
the
enh/hvpa-alternative-for-apiserver
branch
from
1b8d6c8
to
95488be
Compare
gardener-prow
bot
added
cla: no
ialidzhikov
force-pushed
the
enh/hvpa-alternative-for-apiserver
branch
from
95488be
to
0bbad7c
Compare
gardener-prow
bot
removed
the
do-not-merge/work-in-progress
ialidzhikov
force-pushed
the
enh/hvpa-alternative-for-apiserver
branch
2 times, most recently
from
6bc5c1d
to
d68b7a3
Compare
ialidzhikov
force-pushed
the
enh/hvpa-alternative-for-apiserver
branch
from
d68b7a3
to
756e37e
Compare
|
@rfranzke @plkokanov @voelzmo could you PTAL? I would like include this PR in 1.95. |
There was a problem hiding this comment.
Just one typo correction.
I don't want to block this just because of the discussions around behavior for alpha.control-plane.scaling.shoot.gardener.cloud/scale-down-disabled and copying over the resources from existing Deployments – if you feel we should merge it this way, we can go ahead.
| if k.values.Autoscaling.ScaleDownDisabled && hpa.Spec.MinReplicas != nil { | ||
| // If scale-down is disabled and the HPA resource exists and HPA's spec.minReplicas is not nil, | ||
| // then minReplicas is max(spec.minReplicas, status.desiredReplcias). | ||
| // When scale-down is disabled, this allows operators to specify a custom value for HPA spec.minReplicas |
There was a problem hiding this comment.
Thanks for explaining the reasoning behind this change. So what exactly do we want as new behavior when alpha.control-plane.scaling.shoot.gardener.cloud/scale-down-disabled is set? Previously, we had HPA effectively turned off, because minReplicas == maxReplicas == 4.
Now, we allow our operators to set minReplicas on HPA themselves, still set maxReplicas as 4. Which values do we think make sense here? Given that the whole purpose is to stop scaling down I'm not sure if 1-3 are reasonable values?
You're right: We may or may not choose a different value as maxReplicas in the future, but I'm not sure how that influences our decision to make this change in behavior right now.
At the very least: We should document how this annotation works, how you can interact (modify the HPA's minReplicas, not the Deployment's replicas) and note that horizontal downscaling does happen.
| // - When transitioning from HVPA to HPAAndVPA autoscaling mode, we need to preserve the kube-apiserver container resources | ||
| // to do not cause an unwanted rollout that might be breaking. Otherwise, we would scale down from the potentially | ||
| // high resource requests (set by HVPA) to the initial resource requests in HPAAndVPA mode. | ||
| if deployment != nil { |
There was a problem hiding this comment.
If I understand this correctly, this would mean that we will never get rid of this piece of code, which was only introduced because that's how HVPA worked. To me, this is harder to understand and keep in mind as context for the future than clearly scoping this method to the single purpose of not resetting snowflakes.
Do we already have a dedicated test documenting the desired behavior for snowflakes? Just to make sure that future versions of us don't go in and just remove this method.
| if k.values.Autoscaling.ScaleDownDisabled && hpa.Spec.MinReplicas != nil { | ||
| // If scale-down is disabled and the HPA resource exists and HPA's spec.minReplicas is not nil, | ||
| // then minReplicas is max(spec.minReplicas, status.desiredReplcias). | ||
| // When scale-down is disabled, this allows operators to specify a custom value for HPA spec.minReplicas |
There was a problem hiding this comment.
Technically, shouldn't we also set maxReplicas to be equal to minReplicas, even with the current proposal (note, I don't mean equal to 4, just set maxReplicas to w/e the calculated value of minReplicas is)?
If I set the minReplicas in the HPA to 3, maxReplicas would still be 4. Depending on when reconciliation happens and when desiredReplicas changes there could still be scale down from 4 to 3 replicas even if the annotaiton is set:
- Let's say that
desiredReplicasis 3, then this code setsminReplicasto 3 andmaxReplicasto 4. - Replicas get scaled up to 4.
- Then, no reconciliation happens to set
minReplicasto the new number ofdesiredReplicas(4) for some time. - Some time later (before the next reconciliation occurs), scale down happens and replicas are set back to 3.
|
/lgtm |
|
LGTM label has been added. Git tree hash: 1817a7bb68aa6e97d51263dda8dc28649d648988
|
|
/test pull-gardener-e2e-kind-ha-single-zone |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: andrerun, ialidzhikov, voelzmo The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
gardener-prow
bot
added
the
approved
How to categorize this PR?
/area auto-scaling
/kind enhancement
What this PR does / why we need it:
This PR implements proposal outlined in #9562 to replace HVPA for Shoot kube-apiservers with a native VPA and HPA. For more details, see the proposal described in #9562
Which issue(s) this PR fixes:
Part of #9562
Special notes for your reviewer:
All credits goes to @vlerenc and all the involved people for analysing the data and for creating the proposal how we could drop HVPA in a relatively simple and efficient way with native VPA and HPA resources.
Release note:
A new feature gate named `VPAAndHPAForAPIServer` is introduced to gardenlet. When enabled, the Shoot Kubernetes API Server is scaled simultaneously by VPA and HPA on the same metric (CPU and memory usage). The new feature aims to replace the existing HVPA autoscaling mechanism for the Shoot Kubernetes API server.