Update Skew Docs to Fix Compatible Issues between Controller Clients and the API-Server

[network-charles]

This pull request addresses an issue reported by a user regarding the compatibility of the aggregated apiserver library with Kubernetes v1.28 clusters. After updating to version v1.29 of the aggregated apiserver library, the user encountered failures on the Kubernetes v1.28 clusters, accompanied by the following errors:

W0425 14:25:27.110980       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:27.111035       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:34.571189       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:34.571265       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource
W0425 14:25:39.776616       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:39.776683       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:58.606055       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:58.606080       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource
W0425 14:26:01.846754       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:26:01.846786       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:27.211376       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:27.211394       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:34.498183       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:34.498228       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource
W0425 14:25:35.545134       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:35.545287       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:49.677632       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:49.677657       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:51.325292       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:51.325325       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource

Changes Made

• Updated the skew docs to instruct users not to have controller clients such as kubelet, kube-controller-manager, and kube-scheduler, as well as aggregated apiservers of a higher version than the kube-apiserver they interact with.

References

kubernetes/kubernetes#124533
kubernetes/kubernetes#124655

[netlify]

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	906570d
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/663950132f482c00083c4738
😎 Deploy Preview	https://deploy-preview-46109--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[dims]

/assign @liggitt

[liggitt]

I agree with @liggitt, this document isn't the right place to document this. I will be closing the PR.

I think we can mention in this doc that other components have their own skew requirements and their documentation should be consulted to determine what kube-apiserver versions they support and whether they should be upgraded before / after kube-apiserver.

Separately, the aggregated API servers in various sig projects (metrics-server? custom-metrics-server?) should document their supported kube-apiserver versions. It looks like https://github.com/kubernetes-sigs/metrics-server?tab=readme-ov-file#compatibility-matrix and https://github.com/kubernetes-sigs/custom-metrics-apiserver?tab=readme-ov-file#compatibility need updates.

[network-charles]

Separately, the aggregated API servers in various sig projects (metrics-server? custom-metrics-server?) should document their supported kube-apiserver versions. It looks like https://github.com/kubernetes-sigs/metrics-server?tab=readme-ov-file#compatibility-matrix and https://github.com/kubernetes-sigs/custom-metrics-apiserver?tab=readme-ov-file#compatibility need updates.

Okay, Jordan.

I think we can mention in this doc that other components have their own skew requirements and their documentation should be consulted to determine what kube-apiserver versions they support and whether they should be upgraded before / after kube-apiserver.

The skew documentation contains kube-apiserver version requirements for each component. If there's another documentation, kindly let me know.

[liggitt]

The skew documentation contains kube-apiserver version requirements for each component

This page documents supported skew of core kubernetes components. Mentioning in this doc that supported skew of other components should be considered might prompt admins to consider a category of compatibility that they are not considering today.

Separately, the places where kubernetes sigs are documenting compatibility of aggregated API servers they release should be updated to be explicit about the supported kube-apiserver skew. Issues tracking this request in each repo and referencing this discussion would be a good start. Examples:

• https://github.com/kubernetes-sigs/metrics-server?tab=readme-ov-file#compatibility-matrix
• https://github.com/kubernetes-sigs/custom-metrics-apiserver?tab=readme-ov-file#compatibility

[network-charles]

Mentioning in this doc that supported skew of other components should be considered might prompt admins to consider a category of compatibility that they are not considering today.

That's true, I will add it.

Separately, the places where kubernetes sigs are documenting compatibility of aggregated API servers they release should be updated to be explicit about the supported kube-apiserver skew. Issues tracking this request in each repo and referencing this discussion would be a good start. Examples:

https://github.com/kubernetes-sigs/metrics-server?tab=readme-ov-file#compatibility-matrix
https://github.com/kubernetes-sigs/custom-metrics-apiserver?tab=readme-ov-file#compatibility

Great.

[tengqm]

Thanks.
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tengqm

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• content/en/releases/OWNERS [tengqm]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sftim]

Thanks

We prefer headings in sentence case
/remove-area release-eng

[sftim]

I recommend cancelling approval until the hold from #46109 (comment) is addressed. Then we can LGTM and re-approve.

[network-charles]

Thanks for the review, @sftim, the outcome was addressed here and here.