Possible old bug - getting reports of globalmount does not exist sporadically.

[maprager]

What happened:
Trying to mount a known drive occasionally fails with the message

I0122 16:14:54.715136 1 nodeserver.go:79] NodePublishVolume: mounting /var/lib/kubelet/plugins/kubernetes.io/csi/pv//globalmount at /var/lib/kubelet/pods/65bba3c3-bdcb-4074-877a-ac23c575e5fc/volumes/kubernetes.iocsi//mount with mountOptions: [bind] volumeID(///

##)
I0122 16:14:54.715165 1 mount_linux.go:220] Mounting cmd (mount) with arguments ( -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv//globalmount /var/lib/kubelet/pods/65bba3c3-bdcb-4074-877a-ac23c575e5fc/volumes/kubernetes.iocsi//mount)
E0122 16:14:54.716676 1 mount_linux.go:232] Mount failed: exit status 32
Mounting command: mount
Mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/pv//globalmount /var/lib/kubelet/pods/65bba3c3-bdcb-4074-877a-ac23c575e5fc/volumes/kubernetes.iocsi//mount
Output: mount: /var/lib/kubelet/pods/65bba3c3-bdcb-4074-877a-ac23c575e5fc/volumes/kubernetes.iocsi//mount: special device /var/lib/kubelet/plugins/kubernetes.io/csi/pv//globalmount does not exist.

What you expected to happen:
I expected the volume to be mounted - and the pod to start.

How to reproduce it:
Unknown - sometimes it works, and sometimes it doesn't

Anything else we need to know?:

1. Other pods on the same node - seem to be able to access this pv - ( sometimes )
2. K8s Version 1.21
3. Baremetal K8s
4. Driver info:
   I1119 13:12:35.354327 1 smb.go:93]
   DRIVER INFORMATION:

---

Build Date: "2023-09-11T23:25:57Z"
Compiler: gc
Driver Name: smb.csi.k8s.io
Driver Version: v1.13.0
Git Commit: ""
Go Version: go1.20.5
Platform: linux/amd64

Streaming logs below:
I1119 13:12:35.401233 1 mount_linux.go:284] Detected umount with safe 'not mounted' behavior
I1119 13:12:35.401925 1 driver.go:93] Enabling controller service capability: CREATE_DELETE_VOLUME
I1119 13:12:35.401945 1 driver.go:93] Enabling controller service capability: SINGLE_NODE_MULTI_WRITER
I1119 13:12:35.401949 1 driver.go:93] Enabling controller service capability: CLONE_VOLUME
I1119 13:12:35.401968 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_WRITER
I1119 13:12:35.401973 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_READER_ONLY
I1119 13:12:35.401976 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER
I1119 13:12:35.401979 1 driver.go:112] Enabling volume access mode: SINGLE_NODE_MULTI_WRITER
I1119 13:12:35.401982 1 driver.go:112] Enabling volume access mode: MULTI_NODE_READER_ONLY
I1119 13:12:35.401987 1 driver.go:112] Enabling volume access mode: MULTI_NODE_SINGLE_WRITER
I1119 13:12:35.401990 1 driver.go:112] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I1119 13:12:35.401994 1 driver.go:103] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I1119 13:12:35.401999 1 driver.go:103] Enabling node service capability: SINGLE_NODE_MULTI_WRITER
I1119 13:12:35.402001 1 driver.go:103] Enabling node service capability: VOLUME_MOUNT_GROUP
I1119 13:12:35.402005 1 driver.go:103] Enabling node service capability: GET_VOLUME_STATS
I1119 13:12:35.403111 1 server.go:118] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}
I1119 13:12:35.969227 1 utils.go:76] GRPC call: /csi.v1.Identity/GetPluginInfo
I1119 13:12:35.969256 1 utils.go:77] GRPC request: {}
I1119 13:12:35.975036 1 utils.go:83] GRPC response: {"name":"smb.csi.k8s.io","vendor_version":"v1.13.0"}
I1119 13:12:36.063644 1 utils.go:76] GRPC call: /csi.v1.Identity/GetPluginInfo
I1119 13:12:36.063671 1 utils.go:77] GRPC request: {}
I1119 13:12:36.063755 1 utils.go:83] GRPC response: {"name":"smb.csi.k8s.io","vendor_version":"v1.13.0"}
I1119 13:12:36.599218 1 utils.go:76] GRPC call: /csi.v1.Node/NodeGetInfo
I1119 13:12:36.599236 1 utils.go:77] GRPC request: {}
I1119 13:12:36.599285 1 utils.go:83] GRPC response: {"node_id":"xxxxx"}
I1119 13:21:23.734807 1 utils.go:76] GRPC call: /csi.v1.Node/NodeStageVolume

[maprager]

please note: if I create the globablmount directory in the node manually - the mount continues, and the pod works -

kubectl exec -n kube-system -it csi-smb-node-v557q -c smb -- mkdir -p /var/lib/kubelet/plugins/kubernetes.io/csi/pv//globalmount

I found this solution here: #302

[juanchovelezpro]

I'm also getting that error sporadically.

As @maprager mentioned, sometimes it works and sometimes it doesn't.

[andyzhangx]

what's the volumeHandle value of your PVs? could you make sure there is no conflict of those volumeHandle values in all of your PVs?

[sant0s12]

Same here. Seems like the directory is not created.

[jjouanno-kwaku-it]

same issue here.

I create manually folder has workaround mentioned but the plugin delete it and I can't start my pod