Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does fancing work with control-plane nodes too? #22

Open
tumurzakov opened this issue Jan 4, 2022 · 2 comments
Open

Does fancing work with control-plane nodes too? #22

tumurzakov opened this issue Jan 4, 2022 · 2 comments

Comments

@tumurzakov
Copy link

I manage bare metal cluster and suffer from downtime when nodes failing

  1. Worker node failed - status become NotReady but traffic still come to the pods that was on this node. Often this leads to downtime if there was critical components. Now I need to delete this node from cluster to reshedule pods. Nothing special for your project.
  2. Control plane node failed - that was only one time. I have 3 control plane nodes. After one node failed i immediatly deleted it from cluster. But nothing works because kubedns stoped resolving internal names. As I understand this happend because remaing etcd daemons didn't reach an agreement who become a master. Only systemctl restart docker on one of two working control plane nodes resolves an issue. So, could kube-fencing be used on control-plane nodes as well?
@kvaps
Copy link
Owner

kvaps commented Jan 6, 2022

Do you have coredns pods running on single node?

@tumurzakov
Copy link
Author

No,

# kubectl -n kube-system get pods -o wide|grep coredns
coredns-758cc77499-crbmv                1/1     Running   0          22d     10.244.0.4       kube-master-3   <none>           <none>
coredns-758cc77499-z2flm                1/1     Running   0          40d     10.244.7.16      kube-master-2   <none>           <none>

I have 3 master nodes, but coredns deployment contain 2 replicas. Something wrong in configuration?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants