Volume and Pod affinity

[dberardo-com]

i have a scenario with 3 nodes and 1 volume. this one volume has 2 replicas with least-effort auto balancing. this ensures that the 2 replicas will live on different nodes, thus reaching an HA situation.

Now i am stuck with the issue whereby a Pod which needs to access the longhorn volume (via a PVC) spawns on the single node which has no volume replica on it ... this pod gets stuck in Pending state because it cannot attach the volume on its node.

how can i prevent such kind of situations?

please note that using node affinity is not a valid solution, since i cannot know in advance which nodes are being selected by longhorn as host for the 2 replicas.

is it possible to use some kind of affinity rule to avoid this scenarios ? i have tried with data locality "strict-local" but this makes me lose the HA property of the volume, which is indeed a must have

[derekbit]

this pod gets stuck in Pending state because it cannot attach the volume on its node.

The pod should run on the volume attached node. Can you see why the pod isn't running on the node where the volume is attached in your setup from longhron-manager logs?

[dberardo-com]

i see, so if i understand you well, what you are saying is that the pod should never be scheduled on a node that has no access to that specific volume/replica, right ? so basically this is some sort of bug somewhere, could that be ?

also: could it be that the pod was scheduled on a node, but then while restarting, the whole system crashed down and longhorn moved the replica to another node, then when the node came back online, the pod was still scheduled on that node and thus got stuck without a volume ? i am telling this because we indeed had a situation of instability somewhere in the cluster, so this might have been the case

[derekbit]

i see, so if i understand you well, what you are saying is that the pod should never be scheduled on a node that has no access to that specific volume/replica, right ? so basically this is some sort of bug somewhere, could that be ?

Is the volume already attached?
Or can you generate a support bundle and tell us what the problematic volume is?

[dberardo-com]

let me try to replicate somehow, then i will provide the log.

can you confirm that this supposition is correct:

i see, so if i understand you well, what you are saying is that the pod should never be scheduled on a node that has no access to that specific volume/replica, right?

meaning that this sort of situation should already be handled by longhorn without any extra configuration, right ?

[derekbit]

i see, so if i understand you well, what you are saying is that the pod should never be scheduled on a node that has no access to that specific volume/replica, right?

For a RWO volume, a Pod should be scheduled to the volume attached node, but the replicas of the volume can be on different nodes.

[dberardo-com]

For a RWO volume, a Pod should be scheduled to the volume attached node, but the replicas of the volume can be on different nodes.

wow ... that was not clear to me at all ... so basically although replicas may reside on nodes A and B, the pod running on C can still use RWO volumes, even if the data is not available locally ? this means than that everytime pod C needs to access data, there needs to be some network transfer of the data over-the-wire, is this correct ?

is it possible to avoid this without using "strict-local" data locality ? i basically dont want to increase network activity too much, and want pod C to start on node A or B, where replicas are physically available, without loosing high availability (strict-local).

again, i cannot use nodeaffinity rule because i dont know in advance where the replicas will be placed by longhorn. i only care that they are 2, to mantain HA

[dberardo-com]

Is the volume already attached?

@derekbit i can confirm that i was able to reproduce the issue by having the following situation:

pod 1 was running on node A while using volume V mounted on that same node. the cluster has 3 nodes and the volume V is replicated on node B. now the node A was manually rebooted and the pod 1 was moved by k8s on node B. However from the longhorn UI i can see that volume V was still attached to node A, so the pod 1 got stucked on pending forever ... even after force deleting the pod 1 manually, k8s was still rescheduling it on node B and so it was never able to attach the volume ...

the only way i have found to start the pod was to manually detach the volume V from node A and attach it to node B using the GUI ...

can this be avoided ?

---

btw i have dumped the logs of both longhorn-managers on both nodes and i am going to remove some sensitive info and post them here if needed