CSI Driver 2.5.4: error "failed to get shared datastores in kubernetes cluster"

[jsoule6]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

What happened:

Installed the CSI driver and Cloud Controller Manager on a K8s cluster running on vSphere VMs. Everything installed well and it is connecting to vSphere and successfully getting node information. However, when we try to deploy a pod on the cluster with the new Storage Class that we created, we are getting the following error:

failed to get shared datastores in kubernetes cluster. Error: no shared datastores found for nodeVm.

We have a single vCenter and are not using a topology aware setup. We have checked the permissions on the vSphere side for the account that we are using and all looks good. The only thing we can think of that might be causing this is that the host that the Control Plane node is on does not have access to the same datastore that the Worker nodes all do. We tried applying and using a Storage Policy as well, but with the same result.

Is it a requirement that all nodes including the Control Plane have access to at least one Shared datastore?

What you expected to happen:

I would expect the PVC to be created

How to reproduce it (as minimally and precisely as possible):

Create a K8s cluster with CSI 2.5.4 and the other versions mentioned above.

Anything else we need to know?:

Environment:
Using the following versions:

vSphere: 6.7 Update 3
Kubernetes: 1.21
Cloud Controller Manager: 1.21
CSI Driver: 2.5.4

[divyenpatel]

The only thing we can think of that might be causing this is that the host that the Control Plane node is on does not have access to the same datastore that the Worker nodes all do.

This is the reason the driver is not able to find the shared accessible datastore for all nodes. We should have a datastore accessible to all nodes in the cluster including control plane nodes.

[sba30]

We encountered the same issue, when we deploy our workers to a single Vsphere cluster using VSAN storage it works fine, but when we split the workers to be deployed across 2 vsphere clusters, each cluster with their own VSAN storage we get the same error when creating the PVC.

Is there a way in this setup for the PVC to only go to 1 of the vsphere clusters and its VSAN storage?

in our setup its now possible for the 2 vsphere clusters to have shared storage, they each have their own VSAN Storage

[divyenpatel]

@sba30 you can define topology on the nodes, and utilize volume topology feature to provision volume on specific vSphere cluster.
https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/3.0/vmware-vsphere-csp-getting-started/GUID-162E7582-723B-4A0F-A937-3ACE82EAFD31.html

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[divyenpatel]

/remove-lifecycle rotten