CreateVolume: Operation not permitted

[Zyplonox]

Hi i just deployed ceph-csi via helm which works great with the admin user of ceph but with a custom user i get Operation not permitted when the provisioner tries to create the Volume:

root@kube1:~/ceph-csi# kubectl logs deploy/ceph-csi-cephfs-provisioner -n ceph-csi-cephfs --tail 9
Found 3 pods, using pod/ceph-csi-cephfs-provisioner-5b9d856485-xv29s
Defaulted container "csi-cephfsplugin" out of: csi-cephfsplugin, csi-provisioner, csi-snapshotter, csi-resizer, liveness-prometheus
I0425 14:22:35.529471       1 utils.go:198] ID: 186 Req-ID: pvc-132a9388-26be-4f1c-af29-aa23023e1c8d GRPC call: /csi.v1.Controller/CreateVolume
I0425 14:22:35.529686       1 utils.go:199] ID: 186 Req-ID: pvc-132a9388-26be-4f1c-af29-aa23023e1c8d GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-132a9388-26be-4f1c-af29-aa23023e1c8d","parameters":{"clusterID":"","csi.storage.k8s.io/pv/name":"pvc-132a9388-26be-4f1c-af29-aa23023e1c8d","csi.storage.k8s.io/pvc/name":"csi-cephfs-pvc","csi.storage.k8s.io/pvc/namespace":"default","fsName":"kubernetes","volumeNamePrefix":"poc-k8s-"},"secrets":"","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
E0425 14:22:35.534306       1 utils.go:203] ID: 186 Req-ID: pvc-132a9388-26be-4f1c-af29-aa23023e1c8d GRPC error: rpc error: code = Internal desc = rados: ret=-1, Operation not permitted

I tried it with the user capabilities from here: https://github.com/ceph/ceph-csi/blob/devel/docs/capabilities.md#cephfs

mgr "allow rw"
osd "allow rw tag cephfs metadata=kubernetes, allow rw tag cephfs data=kubernetes"
mds "allow r fsname=kubernetes path=/volumes, allow rws fsname=kubernetes path=/volumes/csi"
mon "allow r fsname=kubernetes"

Do i miss some capabilities?

Thanks

Environment details

• Image/version of Ceph CSI driver : v3.11.0
• Helm chart version : 3.11.0
• Kernel version : 6.1.0-20-amd64 (k8s) 5.8.0-63-generic (ceph)
• Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its
  krbd or rbd-nbd) :
• Kubernetes cluster version : v1.28.2
• Ceph cluster version : ceph version 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable)

[Zyplonox]

Updating to ceph 17.2.7 solved the problem for us.