Custom topology keys for CSI nodes

[Kavinraja-G]

I have configured the allowedTopologies for the StorageClasses created by the topolvm. In order to make it work, I need to add those custom topologyKeys in the CSINodes as well.

By deafult, it adds the hostName to the Keys, but I need to add the customKeys/Labesl to them. Is there any workaround to achieve this?

Scenario is I have certains nodes with custom labels, say nodePoolType: stateful which has special disks that are to be consumed by topolvm.

[daichimukai]

Hi @Kavinraja-G,

Scenario is I have certains nodes with custom labels, say nodePoolType: stateful which has special disks that are to be consumed by topolvm.

We are using TopoLVM in a similar environment. In your words, we just deploy topolvm-node (and lvmd) to the nodes with the label nodePoolType: stateful.

I have configured the allowedTopologies for the StorageClasses created by the topolvm. In order to make it work, I need to add those custom topologyKeys in the CSINodes as well.

We don't set allowedTopoloogies for the storage classes. CSINodes are automatically configured by the sidecar container (node-driver-registrarer) of topolvm-node. This setup is working well for us.

I hope this will help.

[Kavinraja-G]

@daichimukai Can you help me understanding the use of CSIStorageCapacity or StorageCapacityTracking over the scheduler extender?

[daichimukai]

As far as I know, topolvm-scheduler (scheduler extender for TopoLVM) does nothing with CSIStorageCapacity.

[dee-kryvenko]

@daichimukai the CSIStorageCapacity that topolvm creates for individual nodes when storageCapacityTracking is enabled has this:

nodeTopology:
  matchLabels:
    topology.topolvm.io/node: ip-10-221-135-172

It does not include any other labels that the StorageClass may possibly have, which are required for Cluster Autoscaler to be able to scale nodepools from 0.

As a result, topolvm fails to provision a volume with:

  Warning  ProvisioningFailed    3m13s                 topolvm.io_topolvm-operator-controller-b7dfb99fc-jlzsj_2e24b7f3-b6f1-4baf-826b-3c0c1a96714c  failed to provision volume with StorageClass "my-sc": error generating accessibility requirements: topology map[topology.topolvm.io/node:ip-10-221-135-172] from selected node "ip-10-221-135-172" is not in requisite: [map[foo:bar] map[foo:baz]]

A storage class looks something like:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: my-sc
parameters:
  csi.storage.k8s.io/fstype: ext4
  topolvm.io/device-class: ssd-thin
provisioner: topolvm.io
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: foo
    values:
    - bar
    - baz

What the topolvm users need is a configuration that would allow to customize what goes into nodeTopology.matchLabels on the CSIStorageCapacity besides topology.topolvm.io/node.

[dee-kryvenko]

A similar feature: https://github.com/openebs/lvm-localpv/blob/c97c4a72178d120390e1c77bd2269181ce37d049/docs/faq.md#1-how-to-add-custom-topology-key

[daichimukai]

Thank you for information. If you want that feature, please open an issue.