Local Replica fails with message: "Local Replica Scheduling Failure Error Message: tags not fulfilled"

[urbaman]

Hi,

I have a strage behivour on some pods.

I see 4 replicas instead of 3, on of wich is "a ghost":

Replicas are on nodes 1,2,4
The pod is scheduled (and running) on node 5 (where there's no proper replica.

The pod is properly running, as if using a proper replica.

How to properly debug and solve this problem?

Thank you.

[mantissahz]

Could you set the setting log-level to Debug or Trace and then delete the extra replica to check if it would still replenish an extra replica? If yes, could you generate the support bundle for the investigation?

[urbaman]

Hi @mantissahz

How to see if the log-level has been deployed?

• Yes, deleting the ghost replica gets a new one created, see no log on the dashboard (do see the support bundle creation logs)
• Longhorn version 1.5.3 on Kubeadm 1.28.4
• How to share a big support bundle?

Thank you.

[mantissahz]

How to see if the log-level has been deployed?

Just modify the setting log-level and you should be able to see [Debug] logs in the longhorn-manager or other Longhorn pods.

How to share a big support bundle?

Could you send it to longhorn-support-bundle@suse.com?
Thanks.

[urbaman]

Update:

I actually have 3 workloads with the "Local Replica Scheduling Failure Error Message: tags not fulfilled" error:

1 pod with just a scheduling failure flag, no ghost replica

1 pod with the scheduling replica and just a "no data locality" flag

1 pod with both scheduling flag and data locality flag

[urbaman]

Hi,

Support Bundle sent (as a Gdrive link).

PS: the third pod above does not have the same scheduling flag, but does have the ghost replica.

[urbaman]

Hi @mantissahz

I see that the logs from the longhorn-manager of the node(s) involved says:

Pod/pvc with both no data locality and not schedulable

There's no available disk for replica pvc-5a3cfa97-fd0d-4fc0-b040-b004c43b0807-r-4d7062d4
Failed to schedule replica of volume with HardNodeAffinity = k8w5"
Ignoring recurring job labels on Volume pvc-5a3cfa97-fd0d-4fc0-b040-b004c43b0807 PVC teleport-kube-agent-teleport-data-teleport-kube-agent-0 due to missing source label

Pod/pvc with no data locality

Failed to schedule replica of volume with HardNodeAffinity = k8w5"

Pod/pvc not schedulable

Ignoring recurring job labels on Volume pvc-0ea4a7ea-f28a-4879-b153-2eb4924900dc PVC uptime-kuma-pvc due to missing source label

[urbaman]

Another insight:

If I set replicas=0 in the statefulset of a pod with no data locality, the ghost replica disappears.
If I set replocas back to 1:

• if the pod gets created on a node with a replica, everything's good.
• if the pod gets created on a node without a replica, the ghost replica appears again.

[urbaman]

@mantissahz found out that, for the "no data locality" problem, I had a label mismatch on node/disk.

I have 6 nodes, each with two disks (a virtual disk and a physical nvme disk, for a two-tier storage system).
Each node is labeled "storage", virtual disks are labeled "vdisk" and nvme disks are labeled "nvme".
I have two storage classes, one for each storage tier.

Found out that the virtual disk on node5 was labeled "nvme", as soon as I re-labeled it "vdisk" the "no data locality" problem vanished.

Also, there are no more "ghost" volumes, probably those where due to the inability to create proper volumes of tier "vdisk" on node5 on which both disks were labeled "nvme".

I still have the "Scheduling Failure Local Replica Scheduling Failure Error Message: tags not fulfilled" problem, can't find a way to solve it.

[urbaman]

Hi @mantissahz

I finally deleted the unschedulable volumes, as the "tags not fulfilled" was probably due to the nodes/disks tag mismatch on node5 during first creation, and for some reason they didn't get the schedulable status cleared after the tag mismatch fixing.

Thank you very much for your help.