Instance manager pod does not respect of node taint?

[jwenjian]

I am confused about the instance manager pod behavior of node taint, hope anyone can help :)

I have a single node cluster with longhorn installed, then I add a new taint to the node, e.g. k=v:NoSchedule,then I delete the instance manager pod, based on my understanding, the new instance manager pod cannot be scheduled since it has no corresponding toleration.

But the new pod is scheduled and running normally, just like before.

I tried to delete the longhorn ui pod, the new UI pod is in pending status and cannot be scheduled to the node, which is expected.

Is there any other dark tech for the instance manager pods?

BTW, when I describe the instance manager pod, I just found 2 tolerations, none of them is for the new taint:

QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason   Age   From     Message
  ----    ------   ----  ----     -------
  Normal  Pulled   54m   kubelet  Container image "longhornio/longhorn-instance-manager:v1_20211210" already present on machine
  Normal  Created  54m   kubelet  Created container replica-manager
  Normal  Started  54m   kubelet  Started container replica-manager

[innobead]

This seems an issue for NoSchedule taint effect only if the longhorn manager pod is still alive on the same node.

Because the longhorn manager will try to create the IM pod w/ the specific node name which will supersede the pod scheduling even though there is no toleration configured as the below code.

https://github.com/longhorn/longhorn-manager/blob/9d57b4cdd29b7997eee65b7cea2ad8d6f0f3a19b/controller/instance_manager_controller.go#L1061-L1061

If you try to delete the longhorn manager pod, I think the IM pods will not be created because the longhorn manager pod will not exist as well.

cc @longhorn/qa

[innobead]

ref: #5161

[ejweber]

Hello @jwenjian. After discussing #5161, the team is considering closing it without implementing it. We feel that it is better for instance-manager to ignore the taint on any node where longhorn-manager is also running, as it is critical to Longhorn's operation. We are inclined to keep the current behavior:

• Longhorn-manager will continue to try to run an instance-manager pod on any node where it itself is running.
• If longhorn-manager is forced off a node, instance-manager can be deleted and will not run again.

I see you originally opened this discussion to try to understand how Longhorn was able to keep instance-manager alive (despite the taint). Do you feel strongly that this behavior should be changed, or were you just curious. If you do feel strongly it should be changed, can you explain why?

Thanks!

[jwenjian]

Hi @ejweber , thanks for all the discussions regarding this issue.

The original reason I opened this discussion is for curious and want know why the instance manager pod does not respect node taints(tech detail), and the reason I am facing the "problem" is I want existing longhorn runs on dedicated node only by applying taint to other nodes.

According to your response, based on my understanding, I can achieve this by:

• add taint to nodes that I don't want longhorn runs on
• delete longhorn pods that runs on that nodes
  • For instance manager pod, I need firstly delete longhorn manager pod then delete instance manager pod

Am I understand right?

And for your question, I do not feel strongly it should be changed, just want know how to make my task done (as I described above).

Thanks!

[ejweber]

This should achieve the desired result, though there are additional Longhorn components and resources that should also be cleaned up. In particular, the node.longhorn.io associated with each node you don't want to participate in the cluster should also be deleted after the longhorn-manager pod is deleted, or Longhorn will still think the node should participate. I don't know off the top of my head what problems this might cause, but deleting it will likely help things work better.

I highly recommend following these steps for using a node selector with Longhorn if your are able (instead of a taint approach). They are well tested and supported by the Longhorn team.

[lordunited]

Hello, I have same problem here . according to our business we must taint all of our nodes because we have dedicated nodes with dedicated works , in this situation we use taint to force this dedication.
only instance manager cant handle toleration because it handled by longhorn-manager.
any ideas?

[lordunited]

  tolerations:
  - key: "stage"
    operator: "Equal"
    value: "enabled"
    effect: "NoExecute"
  - key: "longhorn"
    operator: "Equal"
    value: "enabled"
    effect: "NoExecute"
  - key: "production"
    operation: "Equal"
    value: "enabled"
    effect: "NoExecute"