Cross NS quotas

[omniproc]

/kind feature

What happened:

When a ResourceQuota / LimitRange is created it is enforced per namespace.

---
apiVersion: v1
kind: LimitRange              # Limit PVC min/max
metadata:
  name: storagelimits
spec:
  limits:
  - type: PersistentVolumeClaim
    max:
      storage: 50Gi
    min:
      storage: 1Gi
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: storagequota
spec:
  hard:
    persistentvolumeclaims: "50"  # We want to stay way below the current max for vSphere CSI (which is 59 per node VM)
    requests.storage: "500Gi"     # Limit the amount of storage requested from a StorageClass

Those are namespaced resources that affect any StorageClass used for the given namespace

What you expected to happen:
We'd like to implement limits per StorageClass to make sure vSphere backing datastores are not over-provisioned. Or, to put it differently: we'd like to have a method to make sure the effect on the datastore is the same as if ResourceQuota would be calculated cross-namespaces.

How to reproduce it (as minimally and precisely as possible):

1. Create a new StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
volumeBindingMode: WaitForFirstConsumer
metadata:
  name: rwo-dynamic-block-volume
provisioner: csi.vsphere.vmware.com
allowVolumeExpansion: false
parameters:
  storagepolicyname: "foo"
reclaimPolicy: Delete

2. Apply the policies ResourceQuota / LimitRange shown above
3. Create a PV 499Gi in size in NS A
4. Create a PV 499Gi in size in NS B
5. vSphere datastore now has 998Gi storage space allocated although. We can not currently ensure it will never grow bigger then 500Gi.

Anything else we need to know?:

This is probably a feature vSphere admins are very interested it as it enables control of datastore usage which is currently not possible.
Since this is a vSphere CSI specific request I guess it would make most sense to implement it as a parameter of the csi.vsphere.vmware.com StorageClass object. The responsibility of the admin configuring the StorageClass would be to ensure that the total space defined by this parameter of all StorageClass pointing to the same vSphere datastore doesn't go above the vSphere datastore capacity. I do not believe that there should more logic to this since whatever additional check you might implement it would all break when the same backing datastore is used by another K8s cluster. The scope of this really is:

• We assume StorageClass is configured by a different persona then PV/PVC. Usually that persona has access to the backing datastore and is responsible for it's proper function. Let's call this persona "vSphere admin" for the sake of this discussion.
• We want to make sure users of the StorageClass can not consume more then what the "vSphere admin" allowed. Else they could break the datastore, possibly not only affecting their own workload but also other workload that resides on that datastore.

Environment:

• csi-vsphere version: 3.1.2