In response to this:

What would you like to be added?

Part of kubernetes/enhancements#2340

From https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/2340-Consistent-reads-from-cache/README.md#bug-in-etcd-progress-notification

Only recently community discovered a bug https://github.com/etcd-io/etcd/issues/15220 for requesting progress notification. This bug causes a race between sending an event and progress notification with the same revision. Normally we would expect to event to always come first, however due to the bug, progress notification might be sent first. Serving a consistent lists in that situation could result in returning too early and missing the event. The bug was only fixed in v3.4.25 and v3.5.8 (both 9 months old), which is too fresh for Kubernetes to not handle.

Our tests have shown that with the bug manual progress notification cannot be trusted at all. It causes a silent corruption that cannot be automatically detected prior to acting upon the corrupted data. For Beta, we propose to conditionally enable the feature in a way that mitigates the risk of unaware users stumbling on the bug.

We propose to implement a safeguard that will prevent enabling ConsistentListFromCache on etcd version with broken progress notification. During kube-apiserver startup we will verify etcd version by calling Status method and checking etcd patch version. The condition will only pass if all etcd endpoints have etcd version that includes the fix for https://github.com/etcd-io/etcd/issues/15220.

The proposed behavior of the safeguard for Beta:

etcd version could not be acquired or etcd new enough - warn, but continue with whatever the current value of the feature gate is
etcd too old, feature gate not explicitly specified - warn and disable the feature
etcd too old, feature gate explicitly enabled - abort
etcd new enough - use whatever the current value of the feature gate is
The proposed behavior of the safeguard for GA (feature gate locked to true):

etcd version could not be acquired - warn, but continue with the feature enabled
etcd too old - abort
etcd new enough - enable feature
Checking etcd version during start should be a good enough solution as we generally don't expect etcd clusters to downgrade. As of etcd v3.5 it is not an officially supported feature.

For now posted as a "help wanted" issue, however if we cannot hit the timeline for 1.30 code freeze (5th March) I would take it over so we don't miss. Still issue is simple enough that it can be quickly implemented.

/sig api-machinery
/triage accepted
/help

Why is this needed?

Warn users about running on buggy etcd version

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.