Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kubernetes not working with DNS - no communication from pod to outside world #983

Closed
lukaszpy opened this issue Apr 19, 2018 · 7 comments
Closed

Kubernetes not working with DNS - no communication from pod to outside world #983

lukaszpy opened this issue Apr 19, 2018 · 7 comments
Labels
wontfix

Comments

@lukaszpy
Copy link

lukaszpy commented Apr 19, 2018 •
edited

I have probaly problem with kubernetes DNS as my service cannot communicate to outside world (bitbucker.org).
Actually I found this page:
https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

and validate it on my cluster (no minikube):

  zordon@megazord:~$ kubectl exec busybox cat /etc/resolv.conf
   nameserver 10.96.0.10
   search default.svc.cluster.local svc.cluster.local cluster.local
   options ndots:5

and:

    zordon@megazord:~$ kubectl exec -ti busybox -- nslookup kubernetes.default
    Server:    10.96.0.10
    Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
    
    nslookup: can't resolve 'kubernetes.default'
    command terminated with exit code 1

Any idea how can I resolve to be able connect from inside pod to outside world ?

This probably is related to Flannel, as connectivity from image run only by docker is avalible.
Whant to mention that I have run my cluster with this example: https://blog.alexellis.io/kubernetes-in-10-minutes/

I have also modify https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml
and pass argument --iface with my wifi card which has acces to internet but then kube-flannel-ds cannot start
from:

    args:
            - --ip-masq
            - --kube-subnet-mgr

to:

args:
            - --ip-masq
            - --kube-subnet-mgr
            - --iface=wlan0ec5

zordon@megazord:~$ kubectl get pods  -n kube-system
NAME                               READY     STATUS    RESTARTS   AGE
etcd-megazord                      1/1       Running   1          21m
kube-apiserver-megazord            1/1       Running   1          21m
kube-controller-manager-megazord   1/1       Running   1          22m
kube-dns-86f4d74b45-8gh6q          3/3       Running   5          22m
kube-flannel-ds-2wqqr              1/1       Running   1          17m
kube-flannel-ds-59txb              1/1       Running   1          15m
kube-proxy-bdxb4                   1/1       Running   1          15m
kube-proxy-mg44x                   1/1       Running   1          22m
kube-scheduler-megazord            1/1       Running   1          22m
zordon@megazord:~$ kubectl get svc  -n kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   23m

zordon@megazord:~$ kubectl describe service kube-dns -n kube-system
Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       <none>
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         10.244.0.27:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         10.244.0.27:53
Session Affinity:  None
Events:            <none>


zordon@megazord:~$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
I0419 17:40:11.473047       1 dns.go:48] version: 1.14.8
I0419 17:40:11.473975       1 server.go:71] Using configuration read from directory: /kube-dns-config with period 10s
I0419 17:40:11.474024       1 server.go:119] FLAG: --alsologtostderr="false"
I0419 17:40:11.474032       1 server.go:119] FLAG: --config-dir="/kube-dns-config"
I0419 17:40:11.474037       1 server.go:119] FLAG: --config-map=""
I0419 17:40:11.474041       1 server.go:119] FLAG: --config-map-namespace="kube-system"
I0419 17:40:11.474044       1 server.go:119] FLAG: --config-period="10s"
I0419 17:40:11.474049       1 server.go:119] FLAG: --dns-bind-address="0.0.0.0"
I0419 17:40:11.474053       1 server.go:119] FLAG: --dns-port="10053"
I0419 17:40:11.474058       1 server.go:119] FLAG: --domain="cluster.local."
I0419 17:40:11.474063       1 server.go:119] FLAG: --federations=""
I0419 17:40:11.474067       1 server.go:119] FLAG: --healthz-port="8081"
I0419 17:40:11.474071       1 server.go:119] FLAG: --initial-sync-timeout="1m0s"
I0419 17:40:11.474074       1 server.go:119] FLAG: --kube-master-url=""
I0419 17:40:11.474079       1 server.go:119] FLAG: --kubecfg-file=""
I0419 17:40:11.474082       1 server.go:119] FLAG: --log-backtrace-at=":0"
I0419 17:40:11.474087       1 server.go:119] FLAG: --log-dir=""
I0419 17:40:11.474091       1 server.go:119] FLAG: --log-flush-frequency="5s"
I0419 17:40:11.474094       1 server.go:119] FLAG: --logtostderr="true"
I0419 17:40:11.474098       1 server.go:119] FLAG: --nameservers=""
I0419 17:40:11.474101       1 server.go:119] FLAG: --stderrthreshold="2"
I0419 17:40:11.474104       1 server.go:119] FLAG: --v="2"
I0419 17:40:11.474107       1 server.go:119] FLAG: --version="false"
I0419 17:40:11.474113       1 server.go:119] FLAG: --vmodule=""
I0419 17:40:11.474190       1 server.go:201] Starting SkyDNS server (0.0.0.0:10053)
I0419 17:40:11.488125       1 server.go:220] Skydns metrics enabled (/metrics:10055)
I0419 17:40:11.488170       1 dns.go:146] Starting endpointsController
I0419 17:40:11.488180       1 dns.go:149] Starting serviceController
I0419 17:40:11.488348       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0419 17:40:11.488407       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0419 17:40:11.988549       1 dns.go:170] Initialized services and endpoints from apiserver
I0419 17:40:11.988609       1 server.go:135] Setting up Healthz Handler (/readiness)
I0419 17:40:11.988641       1 server.go:140] Setting up cache handler (/cache)
I0419 17:40:11.988649       1 server.go:126] Status HTTP port 8081

zordon@megazord:~$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
I0419 17:44:35.785171       1 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}
I0419 17:44:35.785336       1 nanny.go:94] Starting dnsmasq [-k --cache-size=1000 --no-negcache --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]
I0419 17:44:35.876534       1 nanny.go:119]
W0419 17:44:35.876572       1 nanny.go:120] Got EOF from stdout
I0419 17:44:35.876578       1 nanny.go:116] dnsmasq[26]: started, version 2.78 cachesize 1000
I0419 17:44:35.876615       1 nanny.go:116] dnsmasq[26]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
I0419 17:44:35.876632       1 nanny.go:116] dnsmasq[26]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0419 17:44:35.876642       1 nanny.go:116] dnsmasq[26]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0419 17:44:35.876653       1 nanny.go:116] dnsmasq[26]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0419 17:44:35.876666       1 nanny.go:116] dnsmasq[26]: reading /etc/resolv.conf
I0419 17:44:35.876677       1 nanny.go:116] dnsmasq[26]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0419 17:44:35.876691       1 nanny.go:116] dnsmasq[26]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0419 17:44:35.876701       1 nanny.go:116] dnsmasq[26]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0419 17:44:35.876709       1 nanny.go:116] dnsmasq[26]: using nameserver 127.0.0.53#53
I0419 17:44:35.876717       1 nanny.go:116] dnsmasq[26]: read /etc/hosts - 7 addresses

zordon@megazord:~$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c sidecar
I0419 17:45:06.726670       1 main.go:51] Version v1.14.8
I0419 17:45:06.726781       1 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})
I0419 17:45:06.726842       1 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}
I0419 17:45:06.726927       1 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:33}

Master node:

zordon@megazord:~$ ip -d route
unicast default via 192.168.1.1 dev wlp32s0 proto static scope global metric 600
unicast 10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
unicast 10.244.1.0/24 via 10.244.1.0 dev flannel.1 proto boot scope global onlink
unicast 169.254.0.0/16 dev wlp32s0 proto boot scope link metric 1000
unicast 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
unicast 192.168.1.0/24 dev wlp32s0 proto kernel scope link src 192.168.1.110 metric 600
zordon@megazord:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp30s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether 4c:cc:6a:f8:7e:4b brd ff:ff:ff:ff:ff:ff
3: wlp32s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ec:08:6b:0c:9c:27 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.110/24 brd 192.168.1.255 scope global wlp32s0
       valid_lft forever preferred_lft forever
    inet6 fe80::f632:2f08:9caa:2c82/64 scope link
       valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:32:19:f7:5a brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:32ff:fe19:f75a/64 scope link
       valid_lft forever preferred_lft forever
6: vethf9de74d@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether ba:af:58:a0:4a:74 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::b8af:58ff:fea0:4a74/64 scope link
       valid_lft forever preferred_lft forever
7: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether a6:d1:45:73:c3:31 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::a4d1:45ff:fe73:c331/64 scope link
       valid_lft forever preferred_lft forever
8: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:58:0a:f4:00:01 brd ff:ff:ff:ff:ff:ff
    inet 10.244.0.1/24 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::24f5:4cff:fee9:a32d/64 scope link
       valid_lft forever preferred_lft forever
9: veth58367f89@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
    link/ether 7a:29:e9:c8:bf:3f brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::7829:e9ff:fec8:bf3f/64 scope link
       valid_lft forever preferred_lft forever

Node worker:

zordon@k8s-minion-one:~$ ip -d route
unicast default via 192.168.1.1 dev enp0s25 proto dhcp scope global src 192.168.1.111 metric 100
unicast 10.244.0.0/24 via 10.244.0.0 dev flannel.1 proto boot scope global onlink
unicast 10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1 linkdown
unicast 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
unicast 192.168.1.0/24 dev enp0s25 proto kernel scope link src 192.168.1.111
unicast 192.168.1.1 dev enp0s25 proto dhcp scope link src 192.168.1.111 metric 100
zordon@k8s-minion-one:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 18:03:73:45:75:71 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.111/24 brd 192.168.1.255 scope global enp0s25
       valid_lft forever preferred_lft forever
    inet6 fe80::1a03:73ff:fe45:7571/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:38:3e:a3:94 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:38ff:fe3e:a394/64 scope link
       valid_lft forever preferred_lft forever
4: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
    link/ether 7a:d0:2a:b4:73:43 brd ff:ff:ff:ff:ff:ff
    inet 10.244.1.0/32 scope global flannel.1
       valid_lft forever preferred_lft forever
    inet6 fe80::78d0:2aff:feb4:7343/64 scope link
       valid_lft forever preferred_lft forever
5: cni0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 0a:58:0a:f4:01:01 brd ff:ff:ff:ff:ff:ff
    inet 10.244.1.1/24 scope global cni0
       valid_lft forever preferred_lft forever
    inet6 fe80::7440:12ff:fefa:f55/64 scope link
       valid_lft forever preferred_lft forever

Envirnoment:

  • Cluster master: Ubuntu 17.1.0 (ufw disabled)
  • Cluster node: Ubuntu serwer 16.10 (ufw disabled)
@lukaszpy
Copy link
Author

Found the problem which exists in DNS (ip adresses works ok. Helped me config map for dns:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
data:
  upstreamNameservers: |-
    ["8.8.8.8", "8.8.4.4"]

@parvez84
Copy link

I am stuck at the same exact place , I tried giving google dns and corp dns it still fails. Any suggestions ?

@dperique
Copy link

This same issue happens to us with minikube 0.23.0, Kubernetes 1.7.5, and Ubuntu 18 (bionic). DNS names don't resolve. The workaround for putting upstreamNameservers in the configmap helped. Thanks @lukaszpy !

We still have to figure out a way to tweak our minikube setups to apply this workaround until this gets fixed.

@randyrue randyrue mentioned this issue Jul 18, 2018
Closed
@hokiegeek2
Copy link

hokiegeek2 commented Aug 24, 2018 •
edited

From inside busybox pod after applying upstreamServer config @lukaszpy mentioned above I get the following for external nslookups (minikube v0.28.1 on ubuntu 18.04):

/ # nslookup 8.8.8.8
Server: 10.96.0.10
Address: 10.96.0.10:53

Non-authoritative answer:
8.8.8.8.in-addr.arpa name = google-public-dns-a.google.com

/ # nslookup google.com
Server: 10.96.0.10
Address: 10.96.0.10:53

Non-authoritative answer:
Name: google.com
Address: 172.217.5.238

*** Can't find google.com: No answer

/ #

When I attempt kubernetes.default, I get this:
/ # nslookup kubernetes.default
Server: 10.96.0.10
Address: 10.96.0.10:53

** server can't find kubernetes.default: NXDOMAIN

*** Can't find kubernetes.default: No answer

/ #

@devdattakulkarni devdattakulkarni mentioned this issue Oct 31, 2018
Closed
@theAkito
Copy link

I experience the same problem, though my YAMLs look different than the provided snippet. Can I still apply the workaround somehow?

@stale
Copy link

stale bot commented Jan 26, 2023

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Jan 26, 2023
@hokiegeek2
Copy link

no longer an issue

@stale stale bot closed this as completed Feb 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
wontfix
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@lukaszpy @hokiegeek2 @dperique @theAkito @parvez84