Cannot access Kubernetes Cluster after Master node Dynamic IP changes #108453
Comments
k8s-ci-robot
added
the
needs-sig
|
@hrabhijith: This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
the
needs-triage
|
/sig k8s-infra |
k8s-ci-robot
added
sig/k8s-infra
@neolit123 does kubeadm allows to change the IP? I don't think that can works since certificates are using those IPs right? |
|
The idea behind running the kubeadm init second time is to regenerate the certs with new IP. I have deleted certs before running kubeadm init. I had also tried giving just --advertise-api-address= newIP to kubeadm init. But still it fails. I have no idea about what does kubeadm do second time, after changing IP. I just followed steps which worked before for people in this scenario. |
changing the IP of the CP node is not supported and complicated, you should have used the control-plane-endpoint option with a DNS name instead of --apiserver-advertise-address. see here: and then just change the IP behind the DNS. also related to switching from single CP node to HA:
running the whole kubeadm init twice is not supported without a kubeadm reset first. you could:
other ideas here: this is generally untested and unsupported. /kind support |
k8s-ci-robot
added
the
kind/support
|
@neolit123: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/remove-sig node network k8s-infra |
k8s-ci-robot
added
sig/cluster-lifecycle
How to switch to DNS for control-plane-endpoint with existing cluster using IP address for both advertise-api-address and control-plane-endpoint? Should I edit kubeadm-config configmap in kube-system ? If yes, should I replace IP to DNS name for these two fields? or update DNS name to control-plane-endpoint only and delete advertise-api-address field? After setting DNS for control-plane-endpoint, and generate new certs using that kubeadm-config, I can change the IP of the CP and everything should go fine?? |
|
I created a new cluster using kubeadm
Then I checked /etc/kubernetes contents: All files have IP and not DNS. Only places the DNS is used in this case are, How this argument is helping to solve IP changes of the node? If changed manually, then above issues. |
|
The DNS would at least help worker nodes to not have to rejoin. |
If you already did this step on the CP node and generated new CA, the worker nodes would have to rejoin anyway because they no longer trust this server and they have no client credentials for it. Its actually less work to recreate the cluster. |
|
Thanks for the replies. I will check the backup option you mentioned. |
|
Hello @neolit123 Do kubeadm has a solution on how the cluster can be setup so that it can handle dynamic change I.P address of Master or Worker nodes? As i understand from the thread we can only address for worker nodes but it doesn't supports I.P change of Master nodes. |
|
not being able to handle dynamic IP changes is a wider k8s problem, so not only a kubeadm one. this comment has more info: |
|
Thank you very much @neolit123 for your response! I have one last query, As you mentioned in below link with your suggestion, this can handle dynamic ip change of worker nodes but not applicable for ip change of master nodes. |
|
yes, if the LB DNS name stays the same workers will be fine. |
What happened?
I had a single node (only master) Kubernetes cluster up and running with couple of workloads. The cluster was created using "Kubeadm".
Now, the IP address of the Master node changes due to DHCP or assume that I had to shift the Master node to other network or unplugged the ethernet and connected to a different NIC. (Dual NIC)
The following issues come up:
1. Kubectl access is lost. So I changed IP in kube config.
The connection to the server <newIP>:6443 was refused - did you specify the right host or port?2. Kubelet had old IP
So I edited /etc/kubernetes/kubelet.env to have new IP and daemon-reload and I restarted Kubelet. Kubelet picked new IP, but has following errors. (--hostname--override = node1)
I restarted and also tried re installing docker. No changes.
I tried the following methods to restore the cluster with new IP address.
Method 1: I saved the contents of /etc/kubernetes/kubeadm-config.yaml.
Once the IP changed, I moved the /etc/kubernetes folder to backup and in the "kubeadm-config.yaml" file from last step I changed the old IP to new IP. I stopped kubelet and then I ran,
kubeadm init --ignore-preflight-errors=DirAvailable--var-lib-etcd --config=kubeadm-config.yamlBut got the following error:
I removed the config file from command and again ran,
kubeadm init --ignore-preflight-errors=DirAvailable--var-lib-etcdBut I got the following error:
After some research, few mentioned about "swap" but it is disabled.
few mentioned about re installing docker, no use.
few mentioned about cgroup driver being different in docker and kubelet. Currently docker has systemd and Kubelet was running fine in old IP. So, how can I get that problem now when just the IP changed?
However, I even changed cgroup for docker to cgroupfs. but still Kubeadm init fails after time out.
Method 2: Once the IP changed, I used the below command to change the old IP to new IP in all the files under /etc/kubernetes.
sudo find /etc/kubernetes -type f | sudo xargs sed -i "s/{{OLD_IP}}/{{NEW_IP}}/"The files updated with new IP are:
Then deleted the old apiserver certs from /etc/kubernetes/ssl. (pki folder is empty by default!)
Then I ran the below command to generate new certs.
kubeadm init phase certs apiserver --config=/etc/kubernetes/kubeadm-config.yamlThen daemon-reload and I restarted kubelet and docker. but still has following errors.
What did you expect to happen?
I expected that kubelet and kube-API server could fetch the new IP from config files and I could access the cluster and all the resources are expected to be just fine. (except few issues regarding IP)
Also, kubeadm init would bring the kubelet up.
How can we reproduce it (as minimally and precisely as possible)?
Change the IP address of Kubernetes master node and then try to access the cluster.
Anything else we need to know?
After Method 1, if I change the IP back to the old IP (actually), the Kubelet starts working and I could access and see the cluster resources without any issues.
However, after Method 2, if I revert all the changes to config to old IP and actually connect the old IP and restrat, I could see the same error in kubelet and I could not even access the cluster from the former IP address.
Kubernetes version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6", GitCommit:"d921bc6d1810da51177fbd0ed61dc811c5228097", GitTreeState:"clean", BuildDate:"2021-10-27T17:50:34Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6", GitCommit:"d921bc6d1810da51177fbd0ed61dc811c5228097", GitTreeState:"clean", BuildDate:"2021-10-27T17:44:26Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node1 Ready control-plane,master 22m v1.21.6 Ubuntu 20.04.3 LTS 5.11.0-27-generic docker://20.10.8
Cloud provider
No cloud provider. Installation at the Edge.
OS version
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
Linux 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Install tools
Kubeadm
Container runtime (CRI) and and version (if applicable)
Client: Docker Engine - Community
Version: 20.10.8
API version: 1.41
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:54:27 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.8
API version: 1.41 (minimum version 1.12)
Go version: go1.16.6
Git commit: 75249d8
Built: Fri Jul 30 19:52:33 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Related plugins (CNI, CSI, ...) and versions (if applicable)
No response
/sig k8s-infra
/sig network
/sig node
The text was updated successfully, but these errors were encountered: