Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ignored pod-eviction-timeout settings #74651

Closed
danielloczi opened this issue Feb 27, 2019 · 17 comments
Closed

ignored pod-eviction-timeout settings #74651

danielloczi opened this issue Feb 27, 2019 · 17 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@danielloczi
Copy link

What happened: I modified the pod-eviction-timeout settings of kube-controller-manager on the master node (in order to to decrease the amount of time before k8s re-creates a pod in case of node failure). The default value is 5 minutes, I configured 30 seconds. Using the sudo docker ps --no-trunc | grep "kube-controller-manager" command I checked that the modification was successfully applied:

kubeadmin@nodetest21:~$ sudo docker ps --no-trunc | grep "kube-controller-manager"
387261c61ee9cebce50de2540e90b89e2bc710b4126a0c066ef41f0a1fb7cf38   sha256:0482f640093306a4de7073fde478cf3ca877b6fcc2c4957624dddb2d304daef5                         "kube-controller-manager --address=127.0.0.1 --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf --client-ca-file=/etc/kubernetes/pki/ca.crt --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --use-service-account-credentials=true --pod-eviction-timeout=30s" 

I applied a basic deployment with two replicas:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
      - image: busybox
        command:
        - sleep
        - "3600"
        imagePullPolicy: IfNotPresent
        name: busybox
      restartPolicy: Always

The first pod created on the first worker node, the second pod created on the second worker node:

NAME         STATUS   ROLES    AGE   VERSION
nodetest21   Ready    master   34m   v1.13.3
nodetest22   Ready    <none>   31m   v1.13.3
nodetest23   Ready    <none>   30m   v1.13.3

NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE   IP          NODE         NOMINATED NODE   READINESS GATES
default       busybox-74b487c57b-5s6g7             1/1     Running   0          13s   10.44.0.2   nodetest22   <none>           <none>
default       busybox-74b487c57b-6zdvv             1/1     Running   0          13s   10.36.0.1   nodetest23   <none>           <none>
kube-system   coredns-86c58d9df4-gmcjd             1/1     Running   0          34m   10.32.0.2   nodetest21   <none>           <none>
kube-system   coredns-86c58d9df4-wpffr             1/1     Running   0          34m   10.32.0.3   nodetest21   <none>           <none>
kube-system   etcd-nodetest21                      1/1     Running   0          33m   10.0.1.4    nodetest21   <none>           <none>
kube-system   kube-apiserver-nodetest21            1/1     Running   0          33m   10.0.1.4    nodetest21   <none>           <none>
kube-system   kube-controller-manager-nodetest21   1/1     Running   0          20m   10.0.1.4    nodetest21   <none>           <none>
kube-system   kube-proxy-6mcn8                     1/1     Running   1          31m   10.0.1.5    nodetest22   <none>           <none>
kube-system   kube-proxy-dhdqj                     1/1     Running   0          30m   10.0.1.6    nodetest23   <none>           <none>
kube-system   kube-proxy-vqjg8                     1/1     Running   0          34m   10.0.1.4    nodetest21   <none>           <none>
kube-system   kube-scheduler-nodetest21            1/1     Running   1          33m   10.0.1.4    nodetest21   <none>           <none>
kube-system   weave-net-9qls7                      2/2     Running   3          31m   10.0.1.5    nodetest22   <none>           <none>
kube-system   weave-net-h2cb6                      2/2     Running   0          33m   10.0.1.4    nodetest21   <none>           <none>
kube-system   weave-net-vkb62                      2/2     Running   0          30m   10.0.1.6    nodetest23   <none>           <none>

To test the correct pod eviction I shutdown the first worker node. After ~1 min the status of the first worker node changed to "NotReady", then
I had to wait +5 minutes (which is the default pod eviction timeout) for pod on the turned off node to be re-created on the other node.

What you expected to happen:
After the node status reports "NotReady", the pod should be re-created on the other node after 30 seconds instead if the default 5 minutes!

How to reproduce it (as minimally and precisely as possible):
Create three nodes. Init Kubernetes on the first node (sudo kubeadm init), apply network plugin (kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"), then join the other two nodes (like: kubeadm join 10.0.1.4:6443 --token xdx9y1.z7jc0j7c8g8lpjog --discovery-token-ca-cert-hash sha256:04ae8388f607755c14eed702a23fd47802d5512e092b08add57040a2ae0736ac).
Add pod-eviction-timeout parameter to Kube Controller Manager on the master node: sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --address=127.0.0.1
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true
    - --pod-eviction-timeout=30s

(the yaml is truncated, only the related first part is showed here).

Check that the settings is applied:
sudo docker ps --no-trunc | grep "kube-controller-manager"

Apply a deployment with two replicas, check that one pod is created on first worker node, the second is created on the second worker node.
Shut down one of the nodes, and check the elapsed time between the event, when the node reports "NotReady" and the pod re-created.

Anything else we need to know?:
I experience the same issue in multi-master environment also.

Environment:

  • Kubernetes version (use kubectl version): v1.13.3
    Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
    Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration: Azure VM
  • OS (e.g: cat /etc/os-release): NAME="Ubuntu" VERSION="16.04.5 LTS (Xenial Xerus)"
  • Kernel (e.g. uname -a): Linux nodetest21 4.15.0-1037-azure Add warnings about self signed certs and MitM attacks. #39~16.04.1-Ubuntu SMP Tue Jan 15 17:20:47 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
  • Others: Docker v18.06.1-ce
@danielloczi danielloczi added the kind/bug Categorizes issue or PR as related to a bug. label Feb 27, 2019
@k8s-ci-robot k8s-ci-robot added the needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Feb 27, 2019
@danielloczi
Copy link
Author

@kubernetes/sig-node-bugs
@kubernetes/sig-apps-bugs

@k8s-ci-robot k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. sig/apps Categorizes an issue or PR as relevant to SIG Apps. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Feb 27, 2019
@k8s-ci-robot
Copy link
Contributor

@danielloczi: Reiterating the mentions to trigger a notification:
@kubernetes/sig-node-bugs, @kubernetes/sig-apps-bugs

In response to this:

@kubernetes/sig-node-bugs
@kubernetes/sig-apps-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@ChiefAlexander
Copy link

I also ran into this issue while testing setting the eviction timeout lower. After poking around at this for sometime I figured out that the cause is the new TaintBasedEvictions.

In version 1.13, the TaintBasedEvictions feature is promoted to beta and enabled by default, hence the taints are automatically added by the NodeController (or kubelet) and the normal logic for evicting pods from nodes based on the Ready NodeCondition is disabled.

Setting the feature flag for this to false causes pods to be evicted like expected. I have not taken time to search through the taint based eviction code but I would guess that we are not utilizing this eviction timeout flag within it.

@ChiefAlexander
Copy link

ChiefAlexander commented Feb 27, 2019

Looking into this more. With TaintBasedEvictions set to true you can set your pods eviction time within its spec under tolerations:
https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#taint-based-evictions
The default values of these are getting set by an admission controller: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/admission/defaulttolerationseconds/admission.go#L34
Those two flags can be set via the kube-apiserver and should achieve the same effect.

@liucimin
Copy link
Contributor

// Controller will not proactively sync node health, but will monitor node
// health signal updated from kubelet. There are 2 kinds of node healthiness
// signals: NodeStatus and NodeLease. NodeLease signal is generated only when
// NodeLease feature is enabled. If it doesn't receive update for this amount
// of time, it will start posting "NodeReady==ConditionUnknown". The amount of
// time before which Controller start evicting pods is controlled via flag
// 'pod-eviction-timeout'.
// Note: be cautious when changing the constant, it must work with
// nodeStatusUpdateFrequency in kubelet and renewInterval in NodeLease
// controller. The node health signal update frequency is the minimal of the
// two.
// There are several constraints:
// 1. nodeMonitorGracePeriod must be N times more than  the node health signal
//    update frequency, where N means number of retries allowed for kubelet to
//    post node status/lease. It is pointless to make nodeMonitorGracePeriod
//    be less than the node health signal update frequency, since there will
//    only be fresh values from Kubelet at an interval of node health signal
//    update frequency. The constant must be less than podEvictionTimeout.
// 2. nodeMonitorGracePeriod can't be too large for user experience - larger
//    value takes longer for user to see up-to-date node health.

@danielloczi
Copy link
Author

Thanks for your feedback ChiefAlexander!
That is the situation, you wrote. I checked the pods, and sure there are the default values assigned to pod for toleration:

kubectl describe pod busybox-74b487c57b-95b6n | grep -i toleration -A 2
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s

So I just simply added my own values to the deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      tolerations:
      - key: "node.kubernetes.io/unreachable"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 2
      - key: "node.kubernetes.io/not-ready"
        operator: "Exists"
        effect: "NoExecute"
        tolerationSeconds: 2
      containers:
      - image: busybox
        command:
        - sleep
        - "3600"
        imagePullPolicy: IfNotPresent
        name: busybox
      restartPolicy: Always

After applying the deployment in case of node failure, node status changes to "NotReady", then pods re-created after 2 seconds.

So we don't have to deal with pod-eviction-timeout anymore, timeout can be set on Pod basis! Cool!

Thanks again for your help!

@nick0323
Copy link

@danielloczi Hi danielloczi , How do you fix this issue? I also meet this issue

@zdyxry
Copy link

zdyxry commented Jun 25, 2019

@323929 I think @danielloczi doesn't care about the pod-eviction-timeout parameter in kube-controller-manager, but solves it by using Taint based Evictions , I tested with Taint based Evictions , it's worked for me.

@danielloczi
Copy link
Author

That is right: I simply started to use Taint based Eviction.

@kamilgregorczyk
Copy link

Is it possible to make it global? I don't want to enable that for each pod config, especially that I use a lot of prepared things from helm

@morgwai
Copy link

morgwai commented Feb 8, 2020

+1 for having the possibility to configure it per whole cluster. tuning per pod or per deployment is rarely useful: in most cases a sane global value is waaay more convenient and the current default of 5m is waaay to long for many cases.

please please reopen this issue.

@richardqa
Copy link

I am facing this same problem, Is there a way to unenable Taint based Evictions and that pod-eviction-timeout works in global mode?

@hrbasic
Copy link

hrbasic commented Mar 30, 2020

I am facing this same problem, Is there a way to unenable Taint based Evictions and that pod-eviction-timeout works in global mode?

I think that you can configure global pod eviction via apiserver: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
I didn't try this, but as I can see there are options: --default-not-ready-toleration-seconds and --default-unreachable-toleration-seconds.

@enricovittorini
Copy link

Why had this bug been marked as closed? It does look like the original issue is not solved, but only work-arounded.
It is not clear to me why the pod-eviction-timeout flag is not working

@pythonzm
Copy link

same issue

@voarsh2
Copy link

voarsh2 commented Apr 18, 2021

I use those lines in deployment - as others say, global/cluster setting is better.
How am I supposed to hit SLA's when it's 5 minutes?

@zhangguanzhang
Copy link

I use those lines in deployment - as others say, global/cluster setting is better.
How am I supposed to hit SLA's when it's 5 minutes?

may you need set this for kube-apiserver :

   --enable-admission-plugins=....,DefaultTolerationSeconds \
  --default-not-ready-toleration-seconds=60 \
  --default-unreachable-toleration-seconds=60 \

ialidzhikov added a commit to gardener/gardener that referenced this issue Dec 25, 2022
… kube-apiserver fields

The kube-controller-manager flag `--pod-eviction-timeout` is deprecated  in favor of the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `--pod-eviction-timeout` flag does not have effect when the taint besed eviction is enabled. The taint  based eviction is beta (enabled by default) since Kubernetes 1.13 and GA since Kubernetes 1.18. For more details, see kubernetes/kubernetes#74651.

This commit allows configuring the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `podEvictionTimeout` field is deprecated in favor of the newly introduced fields. gardener-apiserver no longer defaults the `podEvictionTimeout` field. gardener-apiserver also returns a warning when the `podEvictionTimeout` field is set.
ialidzhikov added a commit to gardener/gardener that referenced this issue Dec 28, 2022
… kube-apiserver fields

The kube-controller-manager flag `--pod-eviction-timeout` is deprecated  in favor of the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `--pod-eviction-timeout` flag does not have effect when the taint besed eviction is enabled. The taint  based eviction is beta (enabled by default) since Kubernetes 1.13 and GA since Kubernetes 1.18. For more details, see kubernetes/kubernetes#74651.

This commit allows configuring the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `podEvictionTimeout` field is deprecated in favor of the newly introduced fields. gardener-apiserver no longer defaults the `podEvictionTimeout` field. gardener-apiserver also returns a warning when the `podEvictionTimeout` field is set.
ialidzhikov added a commit to gardener/gardener that referenced this issue Dec 30, 2022
… kube-apiserver fields

The kube-controller-manager flag `--pod-eviction-timeout` is deprecated  in favor of the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `--pod-eviction-timeout` flag does not have effect when the taint besed eviction is enabled. The taint  based eviction is beta (enabled by default) since Kubernetes 1.13 and GA since Kubernetes 1.18. For more details, see kubernetes/kubernetes#74651.

This commit allows configuring the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `podEvictionTimeout` field is deprecated in favor of the newly introduced fields. gardener-apiserver no longer defaults the `podEvictionTimeout` field. gardener-apiserver also returns a warning when the `podEvictionTimeout` field is set.
ialidzhikov added a commit to gardener/gardener that referenced this issue Dec 30, 2022
… kube-apiserver fields

The kube-controller-manager flag `--pod-eviction-timeout` is deprecated in favor of the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `--pod-eviction-timeout` flag does not have effect when the taint besed eviction is enabled. The taint  based eviction is beta (enabled by default) since Kubernetes 1.13 and GA since Kubernetes 1.18. For more details, see kubernetes/kubernetes#74651.

This commit allows configuring the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `podEvictionTimeout` field is deprecated in favor of the newly introduced fields. gardener-apiserver no longer defaults the `podEvictionTimeout` field. gardener-apiserver also returns a warning when the `podEvictionTimeout` field is set.
ialidzhikov added a commit to gardener/gardener that referenced this issue Jan 13, 2023
… kube-apiserver fields

The kube-controller-manager flag `--pod-eviction-timeout` is deprecated in favor of the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `--pod-eviction-timeout` flag does not have effect when the taint besed eviction is enabled. The taint  based eviction is beta (enabled by default) since Kubernetes 1.13 and GA since Kubernetes 1.18. For more details, see kubernetes/kubernetes#74651.

This commit allows configuring the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `podEvictionTimeout` field is deprecated in favor of the newly introduced fields. gardener-apiserver no longer defaults the `podEvictionTimeout` field. gardener-apiserver also returns a warning when the `podEvictionTimeout` field is set.
ialidzhikov added a commit to gardener/gardener that referenced this issue Jan 13, 2023
… kube-apiserver fields

The kube-controller-manager flag `--pod-eviction-timeout` is deprecated in favor of the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `--pod-eviction-timeout` flag does not have effect when the taint besed eviction is enabled. The taint  based eviction is beta (enabled by default) since Kubernetes 1.13 and GA since Kubernetes 1.18. For more details, see kubernetes/kubernetes#74651.

This commit allows configuring the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `podEvictionTimeout` field is deprecated in favor of the newly introduced fields. gardener-apiserver no longer defaults the `podEvictionTimeout` field. gardener-apiserver also returns a warning when the `podEvictionTimeout` field is set.
ialidzhikov added a commit to ialidzhikov/gardener that referenced this issue Jan 13, 2023
… kube-apiserver fields

The kube-controller-manager flag `--pod-eviction-timeout` is deprecated in favor of the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `--pod-eviction-timeout` flag does not have effect when the taint besed eviction is enabled. The taint  based eviction is beta (enabled by default) since Kubernetes 1.13 and GA since Kubernetes 1.18. For more details, see kubernetes/kubernetes#74651.

This commit allows configuring the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `podEvictionTimeout` field is deprecated in favor of the newly introduced fields. gardener-apiserver no longer defaults the `podEvictionTimeout` field. gardener-apiserver also returns a warning when the `podEvictionTimeout` field is set.
gardener-prow bot pushed a commit to gardener/gardener that referenced this issue Jan 16, 2023
* Allow instantiating v1.26 Kubernetes clients

* Update `README.md` and `docs/usage/supported_k8s_versions.md` for the K8s 1.26

* Maintain Kubernetes feature gates

$ ./hack/compare-k8s-feature-gates.sh 1.25 1.26
Feature gates added in 1.26 compared to 1.25:
APISelfSubjectReview
AggregatedDiscoveryEndpoint
ConsistentHTTPGetHandlers
CrossNamespaceVolumeDataSource
DynamicResourceAllocation
EventedPLEG
LegacyServiceAccountTokenTracking
MinimizeIPTablesRestore
PDBUnhealthyPodEvictionPolicy
PodSchedulingReadiness
StatefulSetStartOrdinal
TopologyManagerPolicyAlphaOptions
TopologyManagerPolicyBetaOptions
TopologyManagerPolicyOptions
ValidatingAdmissionPolicy
WindowsHostNetwork

Feature gates removed in 1.26 compared to 1.25:
CSIMigrationOpenStack
CSRDuration
DefaultPodTopologySpread
DynamicKubeletConfig
IndexedJob
NonPreemptingPriority
PodAffinityNamespaceSelector
PodOverhead
PreferNominatedNode
ServiceLBNodePortControl
ServiceLoadBalancerClass
SuspendJob

Feature gates locked to default in 1.26 compared to 1.25:
CPUManager
CSIMigrationvSphere
DelegateFSGroupToCSIDriver
DevicePlugins
DryRun
EndpointSliceTerminatingCondition
JobTrackingWithFinalizers
KubeletCredentialProviders
MixedProtocolLBService
ServerSideApply
ServiceIPStaticSubrange
ServiceInternalTrafficPolicy
WindowsHostProcessContainers

* Maintain `kube-apiserver` admission plugins

$ ./hack/compare-k8s-admission-plugins.sh 1.25 1.26
Admission plugins added in 1.26 compared to 1.25:
ValidatingAdmissionPolicy

Admission plugins removed in 1.26 compared to 1.25:

* Maintain `ServiceAccount` names for the controllers part of `kube-controller-manager`

$ ./hack/compare-k8s-controllers.sh 1.25 1.26

kube-controller-manager controllers added in 1.26 compared to 1.25:
resource-claim-controller

kube-controller-manager controllers removed in 1.26 compared to 1.25:

* Use 1.26 for local shoot and local e2e test

* Deprecate the `podEvictionTimeout` field in favor of newly introduced kube-apiserver fields

The kube-controller-manager flag `--pod-eviction-timeout` is deprecated in favor of the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `--pod-eviction-timeout` flag does not have effect when the taint besed eviction is enabled. The taint  based eviction is beta (enabled by default) since Kubernetes 1.13 and GA since Kubernetes 1.18. For more details, see kubernetes/kubernetes#74651.

This commit allows configuring the kube-apiserver flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. The `podEvictionTimeout` field is deprecated in favor of the newly introduced fields. gardener-apiserver no longer defaults the `podEvictionTimeout` field. gardener-apiserver also returns a warning when the `podEvictionTimeout` field is set.

* Adapt to the renaming of `etcd_db_total_size_in_bytes` metric to `apiserver_storage_db_total_size_in_bytes`

The metric `etcd_db_total_size_in_bytes` is renamed to `apiserver_storage_db_total_size_in_bytes`. Ref kubernetes/kubernetes#113310.

* Fix the Pod spec in `simple-load-deployment.yaml.tpl`

Test runs of the integration test that uses this template prints the following warning about the issue in the template:
```
{"level":"info","ts":"2022-12-28T19:36:29.043+0200","logger":"KubeAPIWarningLogger","msg":"unknown field \"spec.template.spec.containers[0].nodeName\""}
```

* Update `docs/usage/shoot_credentials_rotation.md`

After the removal of support for Kubernetes < 1.20 Shoot clusters (ref #6987), the kubeconfig Secret no longer has the `token` field. Basic auth cannot be enabled for K8s 1.19+ clusters, hence the kubeconfig Secret cannot contain the `username`/`password` fields anymore.

* Default `enableStaticTokenKubeconfig` to false for Shoots with K8s version >= 1.26

This commit also adapts most of the testmachinery integration tests to use the `shoots/adminkubeconfig` subresource instead of the static kubeconfig.
The Shoot creation intergration is still using the static kubeconfig and it is downloading it to `$TM_KUBECONFIG_PATH/shoot.config`. This commit sets `enableStaticTokenKubeconfig=true` until we figure out which tests/components are using this downloaded kubeconfig.

* Add constraint for K8s version < 1.26

The constraint `ConstraintK8sLess126` is currently not used by gardener/gardener but it is introduced for usage from the extensions.

* Address review comments

* Update `new-kubernetes-version.md` guide with details about the `hyperkube` image

* Update `supported_k8s_versions.md` for K8s 1.26

* Update kube-scheduler component's unit tests for K8s 1.26

* Revert the K8s versions used for e2e tests

For the reasoning, see #7275 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Development

No branches or pull requests