DNS problems on Kubeadm cluster on Ubuntu 22.04

[urbaman]

Hi,

In my environment, I have some Ubuntu 22.04 nodes and a local pfSense DNS serving them.
The /run/systemd/resolve/resolv.conf file is right and I can nslokkup things from the host

nslookup www.google.com
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   www.google.com
Address: 216.239.38.120
Name:   www.google.com
Address: 2001:4860:4802:32::78

But I can't do it from pods on fresh install (Calico CNI):

kubectl exec -i -t dnsutils -- host -a google.com
Trying "google.com.default.svc.cluster.local"
Trying "google.com.svc.cluster.local"
Trying "google.com.cluster.local"
Trying "google.com.urbaman.it"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6694
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com.urbaman.it.         IN      ANY

;; ANSWER SECTION:
google.com.urbaman.it.  30      IN      HINFO   "RFC8482" ""
google.com.urbaman.it.  30      IN      RRSIG   HINFO 13 4 3600 20240207125627 20240205105627 34505 urbaman.it. BwSxfSCJBfWC1ZMzdNV+eMMbQOv/+I0uz+HGAXniK7ZFbCWxXBVvU5bn emrs+Bko7y3b1Fl1kltetpmOdYDnng==

Received 208 bytes from 10.1.0.10#53 in 12 ms

kubectl exec -i -t dnsutils -- nslookup google.com
Server:         10.1.0.10
Address:        10.1.0.10#53

Non-authoritative answer:
*** Can't find google.com: No answer

This causes most pods not reaching outside the cluster/local environment, they can do it only with dnsConfig set to ndots = 1.

How can I solve it?

Thank you.

[chrisohaver]

Your upstream dns is providing a NOERROR answer for “google.com.urbaman.it” hence the client stops searching and does not send a query for “google.com”

One work around can be to craft a custom resolv.conf that does not contain your urbaman.it domain as a search domain, and use that in your kubelet/kubeadm configs.

[urbaman]

I'm confused why it works on the host but not in the pods. Is it because of the other search domains added for in-cluster resolution, going over the default 5 ndots?

Anyway, so I just create a resolv.conf file with just the upstream nameserver(s) and no search domain?

Thank you very much.

[chrisohaver]

In Kubernetes, hosts and pods use different resolv.conf files. The one pods use is constructed by kubelet and contains a list of cluster domains including local domains from the resolv.conf of the pod’s host. You’ll want to override that and tell kubelet to use an alternate resolv.conf - which would be a resolv.conf with your hosts upstreams, but without the private domain.

[chrisohaver]

A search for “kubeadmn resolv.conf” should provide more detailed in depth explanations.

[urbaman]

Works like a charm: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure/#applying-kubelet-configuration-changes

It does not restart workloads as stated, I had to do it manually, and I also loose the feature to get local services resolved without the proper urbaman.it domain, but it works.

Thank you very much!