[Beta: 1.31] Consistent Reads from Cache

[shekhar-rajak]

Enhancement Description

• One-line enhancement description : Consistent Reads from Cache
• Kubernetes Enhancement Proposal:

Kubernetes Get and List requests are guaranteed to be "consistent reads" if the
resourceVersion parameter is not provided. Consistent reads are served from
etcd using a "quorum read".

But often the watch cache contains sufficiently up-to-date data to serve the
read request, and could serve it far more efficiently.

This KEP proposes a mechanism to serve most reads from the watch cache
while still providing the same consistency guarantees as serving the
read from etcd.

• Discussion Link:
• KEP link: https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/2340-Consistent-reads-from-cache/README.md
• Primary contact (assignee): @serathius
• Responsible SIGs: sig-api-machinery
• Enhancement target (which target equals to which milestone):
  • Alpha 1.28
    • KEP (k/enhancements) update PR(s):
      • Revive 2340 Consistent reads from cache KEP #4047
    • Code (k/k) update PR(s): Implement Alpha state for KEP #2340 kubernetes#118508
    • Docs (k/website) update PR(s): Document Consistent Reads from Cache website#42083
    • Scalability tests: Test consistent list from cache test-infra#30094
  • Beta 1.31
    • KEP (k/enhancements) update PR(s):
      • Move KEP 2340 to Beta #4134
    • Code (k/k) update PR(s):
      • Implement checking etcd version to warn about deprecated etcd versions kubernetes#123264

Tasks

Beta Give feedback

1. consistent-list-from-cache tests are broken perf-tests#2540
   help wanted kind/bug +2
2. Implement apiserver_watch_cache_read_wait metric kubernetes#123185
   help wanted kind/feature sig/api-machinery triage/accepted +4
3. Measure apiserver_watch_cache_read_wait metric in scalability tests
4. Implement per-request watch cache opt-out kubernetes#123187
   kind/feature sig/api-machinery triage/accepted +3
   
5. Implement checking etcd version to warn about deprecated etcd versions kubernetes#123192
   help wanted kind/feature sig/api-machinery sig/etcd triage/accepted +5
   
6. Document deprecation of support of etcd v3.3, v3.4.24 and v3.5.7
7. Move ConsistentListFromCache to Beta default kubernetes#123513
   area/apiserver area/cloudprovider area/code-generation area/dependency area/kube-proxy area/kubectl area/kubelet area/provider/gcp area/test cncf-cla: yes do-not-merge/work-in-progress kind/feature needs-priority ok-to-test release-note sig/api-machinery sig/architecture sig/auth sig/cli sig/cloud-provider sig/instrumentation sig/network sig/node sig/storage sig/testing size/M triage/accepted +27

[shekhar-rajak]

#2331 created for migrating the old keps into new template.

[LappleApple]

/sig api-machinery

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[k8s-triage-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[serathius]

Reviving the KEP as agreed on SIG-Apimachinery meeting May 31st 2023. https://youtu.be/sueYXW89ZHg
/assign

PR #4047 to move KEP to implementable

[serathius]

/stage alpha

[serathius]

/milestone v1.28

[jpbetz]

/lead opt-in

[Atharva-Shinde]

Hello @shekhar-rajak @serathius 👋, Enhancements team here.

Just checking in as we approach Enhancements freeze on Thursday, 16th June 2023.

Looks like this enhancement is targeting for stage alpha for v1.28

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone:v1.28
• KEP readme has an updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

There are some nits remaining to be addressed:

• Update kep.yaml with latest last update date.
• Add acknowledgement under the Test Plan section.

I won't decline this KEP for enhancements freeze because of these minor changes but would be great to get a PR ready addressing them.

The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with appropriate stages as well. Thank you :)

[serathius]

Don't understand the unmarked points.

KEP readme using the latest template has been merged into the k/enhancements repo.
KEP readme has an updated detailed test plan section filled out

Tests follows the latest template (maybe roughly) and uses a detailed test plan. Please provide criteria that you used to decide that they are not.

[Atharva-Shinde]

Hey @serathius , the second unmarked point is a subset of the first unmarked point.
As mentioned in my comment above, there is an acknowledgement at the start of the Test Plan section quoting:

[ ] I/we understand the owners of the involved components may require updates to
existing tests to make this code solid enough prior to committing the changes necessary
to implement this enhancement.

which this KEP currently doesn't have and we need that to be included so that the KEP is in compliance with the latest KEP readme template.

[serathius]

Makes sense, suggestion for future. Separate questionnaire to KEP authors (also applies to PRR) from KEPs design document into separate file. Having agreements in the middle of the design makes it less readable, doesn't contribute to overall design and also cannot be automatically validated via ./hack/verify-kep.sh.

[salehsedghpour]

Hello @serathius 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 9th February 2024.

This enhancement is targeting for stage beta for v1.30 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.30.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

The status of this enhancement is marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[kcmartin]

Hi @serathius,

👋 from the v1.30 Communications Team! We'd love for you to opt in to write a feature blog about your enhancement!

We encourage blogs for features including, but not limited to: breaking changes, features and changes important to our users, and features that have been in progress for a long time and are graduating.

To opt in, you need to open a Feature Blog placeholder PR against the website repository.
The placeholder PR deadline is 27th February, 2024.
Here's the 1.30 Release Calendar

[drewhagen]

Hello @serathius 👋, 1.30 Docs Lead here.

Does this enhancement work planned for 1.30 require any new docs or modification to existing docs?
If so, please follows the steps here to open a PR against dev-1.30 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday February 22nd 2024 18:00 PDT.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.
Thank you!

[serathius]

@drewhagen Done kubernetes/website#45239

[salehsedghpour]

Hey again @serathius 👋 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Wednesday 6th March 2024 .

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze (and we need to update the Issue description to include all the related PRs of this KEP):

• Implement checking etcd version to warn about deprecated etcd versions kubernetes#123264

With this, it is now marked as at risk for code freeze for the v1.30 Code Freeze!

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.
As always, we are here to help if any questions come up. Thanks!

[serathius]

@salehsedghpour Did you mean 5th March and not 9th February which is the enhancement freeze?

[salehsedghpour]

Oh You're right, I copied the wrong date. Thanks for noticing! I updated my comment.

[serathius]

@kcmartin done

[salehsedghpour]

Hello @serathius 👋 Enhancements team here,

Unfortunately, the implementation (code related) PR(s) associated with this enhancement is not in the merge-ready state by code-freeze and hence this enhancement is now removed from the v1.30 milestone.

If you still wish to progress this enhancement in v1.30, please file an exception request. Thanks!

/milestone clear

[serathius]

/milestone v1.31

[serathius]

We slipped from v1.30 due to etcd-io/etcd#17507.

Will attempt to land it in v1.31, preliminary validation of v3.5.13 shows that K8s test pass kubernetes/kubernetes#124469.

Next step is to merge kubernetes/kubernetes#124469, kubernetes/kubernetes#124027 and kubernetes/kubernetes#123264

[sreeram-venkitesh]

@serathius Since this KEP is planned for v1.31, I'm keeping the lead-opted-in label. It has already been added to the v1.31 enhancements tracking board. Please let me know if this works.

[serathius]

Great, thanks @sreeram-venkitesh!

[serathius]

The performance comparison of scalability 5k performance tests with consistent list from cache enabled:

| Consistent List From Cache |       | kube-apiserver CPU [cores] | kube-apiserver Memory [MB] | etcd CPU [cores] | etcd Memory [GB] | LIST latency [ms] slowest resource (pods) | LIST latency [ms] second slowest resource        | LIST latency [ms] second third resource        |
| -------------------------- | ----- | -------------------------- | -------------------------- | ---------------- | ---------------- | ----------------------------------------- | ------------------------------------------------ | ---------------------------------------------- |
| Enabled                    | 50ile | 6.305                      | 42.18                      | 1.1              | 4.68             | 200.00                                    | 55.00                                            | 38.00                                          |
|                            | 90ile | 11.99                      | 47.78                      | 1.426            | 5.40             | 1460                                      | 180                                              | 180                                            |
|                            | 99ile | 26.466                     | 49.59                      | 2.1847           | 6.26             | 1490.00                                   | 300.00                                           | 198.00                                         |
| Disabled                   | 50ile | 9.085                      | 43.90                      | 1.5              | 4.84             | 3500                                      | 38                                               | 38                                             |
|                            | 90ile | 17.72                      | 48.94                      | 1.964            | 5.29             | 4850.00                                   | 92.70                                            | 92.70                                          |
|                            | 99ile | 27.132                     | 50.39                      | 2.5547           | 5.93             | 4990.00                                   | 274.00                                           | 272.00                                         |

Improvements summary

• 30% improvement to kube-apiserver CPU usage
• 3 % improvement to kube-apiserver Memory usage
• 25% improvement to etcd CPU usage
• 5s -> 1.5s reduction on 99%ile pods LIST request latency

Some degradations that might require parameter tuning/investigation.

• 3% increase of etcd Memory usage - unexpected
• 90ms -> 180ms increase on 90%ile LIST request latency for resources with no frequent updates. Expected, due to progress notifies being slower than LIST requests in this case.

cc @deads2k @MadhavJivrajani @p0lyn0mial

[dipesh-rawat]

Hello @serathius 👋, 1.31 Enhancements team here.

Just checking in as we approach enhancements freeze on on 02:00 UTC Friday 14th June 2024 / 19:00 PDT Thursday 13th June 2024.

This enhancement is targeting for stage beta for 1.31 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.31. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline so that the PRR team has enough time to review your KEP.

Nit: Update issue description to highlight Beta: 1.31

With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!