Microk8s certificate issue on clean install, loopback interface is not included in certificate

[AllardKrings]

I have an issue with microk8s on a clean install.

Wheb issuing “microk8s kubectl logs “podname” -n “namespace” i get:

Error from server: Get "https://10.0.1.1:10250/containerLogs/kube-system/calico-node-qm4hm/calico-node": tls: failed to verify certificate: x509: certificate is valid for 192.168.2.110, 192.168.2.42, 172.17.0.1, 172.19.0.1, 172.18.0.1, not 10.0.1.1

If i look at the interfaces on the host i get:

br-5d5df39e9204: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.19.0.1 netmask 255.255.0.0 broadcast 172.19.255.255
ether 02:42:de:a0:96:e5 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

br-8ebd9329fc32: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.18.0.1 netmask 255.255.0.0 broadcast 172.18.255.255
ether 02:42:c8:85:0f:bf txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

cali4d2417c4519: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet6 fe80::ecee:eeff:feee:eeee prefixlen 64 scopeid 0x20
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 1174 bytes 115481 (115.4 KB)
RX errors 0 dropped 1 overruns 0 frame 0
TX packets 1143 bytes 627759 (627.7 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

cali9608299badf: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet6 fe80::ecee:eeff:feee:eeee prefixlen 64 scopeid 0x20
ether ee:ee:ee:ee:ee:ee txqueuelen 0 (Ethernet)
RX packets 2004 bytes 194330 (194.3 KB)
RX errors 0 dropped 1 overruns 0 frame 0
TX packets 2052 bytes 205419 (205.4 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
ether 02:42:db:24:32:43 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.110 netmask 255.255.255.0 broadcast 192.168.2.255
inet6 fe80::2e0:4cff:fe01:2774 prefixlen 64 scopeid 0x20
ether 00:e0:4c:01:27:74 txqueuelen 1000 (Ethernet)
RX packets 1431866 bytes 1516803874 (1.5 GB)
RX errors 0 dropped 144548 overruns 0 frame 0
TX packets 423364 bytes 61323721 (61.3 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 10692977 bytes 6776028327 (6.7 GB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 10692977 bytes 6776028327 (6.7 GB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo:microk8s: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 10.0.1.1 netmask 255.255.255.255
loop txqueuelen 1000 (Local Loopback)

vxlan.calico: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.1.16.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::64bc:cdff:fe1b:8317 prefixlen 64 scopeid 0x20
ether 66:bc:cd:1b:83:17 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 49 overruns 0 carrier 0 collisions 0

wlp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.2.42 netmask 255.255.255.0 broadcast 192.168.2.255
inet6 fe80::24a7:5a8c:7a26:ae32 prefixlen 64 scopeid 0x20
ether f8:e4:e3:77:83:fa txqueuelen 1000 (Ethernet)
RX packets 368730 bytes 114797664 (114.7 MB)
RX errors 0 dropped 71665 overruns 0 frame 0
TX packets 10563 bytes 1292726 (1.2 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

It seems that the loopback interface is not included in the certificate.

I am running ubuntu jammy

PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Hardware is a Latte Panda Alpha

[neoaggelos]

Hi @AllardKrings

Unfortunately, this is an issue with the host-access addon. The addon adds a lo:microk8s loopback interface with the IP 10.0.1.1, and it looks like kubelet picks it up for the default node IP.

Two ways around it:

• Do you need the host-access addon? Under normal circumstances, you probably do not, therefore you can just remove it.
• Edit the kubelet arguments file /var/snap/microk8s/current/args/kubelet and add a line with --node-ip=192.168.2.110 (based on your network interfaces output in the description. Then restart microk8s with sudo snap restart microk8s. This way, Kubelet will always use this IP address instead of relying on auto-detection.

[AllardKrings]

Thanks a lot! I disabled host-access as a temporaly fix and will wait until it is fixed. I found another way work-around: use --insecure-skip-tls-verify-backend=true Op ma 1 apr 2024 om 18:26 schreef Angelos Kolaitis ***@***.***

…

Hi @AllardKrings <https://github.com/AllardKrings> Unfortunately, this is an issue with the host-access addon. The addon adds a lo:microk8s loopback interface with the IP 10.0.1.1, and it looks like kubelet picks it up for the default node IP. Two ways around it: - Do you need the host-access addon? Under normal circumstances, you probably do not, therefore you can just remove it. - Edit the kubelet arguments file /var/snap/microk8s/current/args/kubelet and add a line with --node-ip=192.168.2.110 (based on your network interfaces output in the description. Then restart microk8s with sudo snap restart microk8s. This way, Kubelet will always use this IP address instead of relying on auto-detection. — Reply to this email directly, view it on GitHub <#4477 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AET5SGIQEWYNPU6MK6FVU2LY3GDEZAVCNFSM6AAAAABFRELUTKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZQGEYDCOJXHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[minhvn]

Hi @AllardKrings Im have same issue, are you solved that? Can share me solution.
Many thankyou.

tls: failed to verify certificate: x509: certificate is valid for 192.168.20.183, 172.17.0.1, not 10.0.1.1

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 10.0.1.1/32 scope global lo:microk8s
valid_lft forever preferred_lft forever

[AllardKrings]

hello @minhvn ,

I used the work-around describe above:

use --insecure-skip-tls-verify-backend=true in your kubectl command