Supporting multiple kubelet root dirs across a cluster with CephFS in Kubernetes

[Bpmm9012]

Description:

We're running a Kubernetes (k8s) cluster with some machines where each machine has a different kubelet root directory configured as follows:

Machine 1: /container/k8s
Machine 2: /data/k8s
Machine 3: /var/lib/kubelet
The default CephFS configuration examples typically use a single hostPath like this:

Yaml
hostPath:
path: /var/lib/kubelet/...

However, this doesn't account for our multi-root directory setup. How can we configure ceph-csi (specifically for CephFS) to work seamlessly across all nodes considering their differing kubelet root directories?

Question:

What is the recommended approach to ensure that ceph-csi with CephFS can handle multiple kubelet root directories on a 3-node k8s cluster? Are there any best practices or specific configurations within ceph-csi or Kubernetes to accommodate such a scenario?

[Madhu-1]

@Bpmm9012 yes it's possible but it complicates things as we need to have many daemonset templates which will use different kubelet path(s) which will have customer taints/tolerations to tell where the pod should go, IMHO we can document it but it will be difficult to maintain this yaml, @Bpmm9012 have you tried some symlink or something similar?

[Bpmm9012]

@Madhu-1 In appreciation of your explanation, setting up hard symlink seems like an effective approach. I will try to utilize the mount --bind command to made all different directories to the same directory.