Object sync between container mount path on windows node and smb shared path doesn't happen even if volume mount succeeded

[TheCodeCargo]

What happened:

We are using the csi smb on windows node on an onprem kubernetes for storage class and we are mounting pvc to windows container pod. We could see that PV and PVC are in bound state and pod is up and running but still we could see that the sync between the container mount path and smb shared path is missing.
There are 1800 objects on smb share path but on the container mount path there is no objects.

We checked the smb controller logs we could see below error:

2023-08-16T14:23:25.178452100+02:00 E0816 14:23:25.177792 23260 utils.go:81] GRPC error: rpc error: code = Internal desc = Could not mount target "c:\var\lib\kubelet\pods\7a4a66d4-ff25-49d6-99a9-173ca7ad63bd\volumes\kubernetes.iocsi\pvc-fa96da59-bde4-4d8b-82bf-42000ea6eb26\mount": open c:\var\lib\kubelet\pods\7a4a66d4-ff25-49d6-99a9-173ca7ad63bd\volumes\kubernetes.iocsi\pvc-fa96da59-bde4-4d8b-82bf-42000ea6eb26\mount: Access is denied.
2023-08-16T14:24:07.158684000+02:00 E0816 14:24:07.158684 23260 utils.go:81] GRPC error: rpc error: code = NotFound desc = path c:\var\lib\kubelet\pods\7a4a66d4-ff25-49d6-99a9-173ca7ad63bd\volumes\kubernetes.io~csi\pvc-fa96da59-bde4-4d8b-82bf-42000ea6eb26\mount does not exist

Also we could confirm that when we restarted the pod the object sync works well and everything went fine again!

What you expected to happen:

When PVC is mounted successfully, all the objects from shared path and the container mount path should be in sync

How to reproduce it:

Anything else we need to know?:

Environment:

• CSI Driver version: v1.10.0
• Kubernetes version (use kubectl version): 1.26.0+rke2r2
• OS (e.g. from /etc/os-release): windows server 2019 (windows worker node), amd-64
• Kernel (e.g. uname -a):
  Linux EDGEMASTER 5.10.0-23-amd64 test: fix travis config #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux

[TheCodeCargo]

Hi Anyone faced this issue ? Can we have a root cause for this.

[awsitcloudpro]

I am also facing a similar issue with CSI SMB driver version v1.11.0 on K8s v1.27.x, with Windows node OS version server core 2019.
In my case, simply restarting the Windows pods / deployment does not work at all. There is a workaround that works - as mentioned here. i.e. Adding SYNCHRONIZE bit to the SMB user's permissions explicitly using ICACLS - followed by Windows pod / deployment restart - works, but only for few days, and breaks again. Running the ICACLS command periodically in a scheduled job seems to be the only way to make it work for longer time.
The same Microsoft document says "This issue doesn't occur if you disable the SMB2 protocol on the client". Is the CSI SMB driver forcing use of SMB protocol version 2? I tried setting vers=3.0 in mount options, but that didn't help.
This issue does not occur if the Windows pods use hostPath volumes with an SMB Global Mapping of the shared folder on Windows nodes, instead of using CSI SMB PVCs.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[awsitcloudpro]

/remove-lifecycle stale

[awsitcloudpro]

/remove-lifecycle rotten