Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add quota support for PVC with VolumeAttributesClass #124360

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Add quota support for PVC with VolumeAttributesClass #124360

wants to merge 2 commits into from
+400 −1

Conversation

carlory
Copy link
Member

@carlory carlory commented Apr 18, 2024 •
edited

What type of PR is this?

/kind feature

What this PR does / why we need it:

A cluster admin wants to control costs while giving application developers the freedom to optimize their applications. They set a per VolumeAttributesClass limit to the maximum count of PVCs that can be specified in a cluster ResourceQuota. When an application dev modifies a PVC to request a higher tier of VolumeAttributesClass but there is no quota, the request is rejected. As the ResourceQuota is a cluster-scoped object, only the cluster admin and not application devs can change limits.

An example of defining ResourceQuota for VolumeAttributesClass:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: glod-pvcs
spec:
  hard:
    count/persistentvolumeclaims: "10"
  scopeSelector:
    matchExpressions:
    - operator : In
      scopeName: VolumeAttributesClass # Match persistentvolumesclaims that references the specified volume attributes class.
      values: ["glod"]

Which issue(s) this PR fixes:

Fixes #119299

Special notes for your reviewer:

Does this PR introduce a user-facing change?

It introduces a new scope name `VolumeAttributesClass`. 

It matches all PVC objects that have the volume attributes class mentioned. 

If you want to limit the count of PVCs that have a specific volume attributes class. In that case, you can create a quota object with the scope name `VolumeAttributesClass` and a matchExpressions that match the volume attributes class.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [KEP]: https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/3751-volume-attributes-class

@k8s-ci-robot
Copy link
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Apr 18, 2024
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Apr 18, 2024
@k8s-ci-robot k8s-ci-robot added area/test kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Apr 18, 2024
@carlory carlory mentioned this pull request Apr 18, 2024
Draft
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: carlory
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@carlory carlory mentioned this pull request Apr 22, 2024
Open
deads2k
deads2k reviewed May 3, 2024
View reviewed changes
pkg/quota/v1/evaluator/core/persistent_volume_claims.go
Comment on lines +312 to +321
if ptr.Deref(pvc.Spec.VolumeAttributesClassName, "") != "" {
vacNames.Insert(*pvc.Spec.VolumeAttributesClassName)
}
if ptr.Deref(pvc.Status.CurrentVolumeAttributesClassName, "") != "" {
vacNames.Insert(*pvc.Status.CurrentVolumeAttributesClassName)
}
modifyStatus := pvc.Status.ModifyVolumeStatus
if modifyStatus != nil && modifyStatus.TargetVolumeAttributesClassName != "" {
vacNames.Insert(modifyStatus.TargetVolumeAttributesClassName)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we quota spec, but we avoid quota-ing status. The actual state should always be allowed to be written.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if we only quota spec field, we will still have this trouble #124436.

For example,

Firstly, we create 2 quota, one for gold and one for silver.
Secondly, we create a pvc with volumeAttributesClass=gold, the gold quota will be used.
Thirdly, we update the pvc with volumeAttributesClass=silver, the expected behavior is that the silver quota will be used and the gold quota will be released. In fact, no change will happen.

@deads2k

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @xing-yang @sunnylovestiramisu @msau42

Copy link
Member Author

@carlory carlory May 7, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't quota status field, users can create unlimited volumes with special attributes by changing a PVC's vac into a non-existent vac. Those modified PVCs have the same attributes as the original PVCs.

Should we introduce a new resource name for this feature like storageclass instead of using the scope mechanism?

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 10, 2024
@k8s-ci-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deads2k deads2k deads2k left review comments

@andrewsykim andrewsykim Awaiting requested review from andrewsykim

@aojea aojea Awaiting requested review from aojea

@msau42 msau42 Awaiting requested review from msau42

@sunnylovestiramisu sunnylovestiramisu Awaiting requested review from sunnylovestiramisu

@xing-yang xing-yang Awaiting requested review from xing-yang

Assignees
No one assigned
Labels
area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/feature Categorizes issue or PR as related to a new feature. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
SIG Apps
Status: Needs Triage
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

1.30 - [kep-3751] Quota Change
3 participants
@carlory @k8s-ci-robot @deads2k