Coredns waiting on kubernetes indefinitely

[vmarchaud]

Hi ! I'm trying to deploy coredns on a self managed kubernetes cluster (v1.21.4), i'm using the official helm chart (version 1.8.4) with the following values:

service:
    clusterIP: "172.16.255.42"
  serviceAccount:
    create: true
  hpa:
    enabled: true
    minReplicas: 1
    maxReplicas: 2
    metrics: []
  servers:
  - zones:
    - zone: .
    port: 53
    plugins:
    - name: errors
    - name: health
      configBlock: |-
        lameduck 5s
    - name: ready
    - name: kubernetes
      parameters: cluster.local 172.16.0.0/16 in-addr.arpa ip6.arpa
      configBlock: |-
        pods verified
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    - name: prometheus
      parameters: 0.0.0.0:9153
    - name: forward
      parameters: . 1.1.1.1 9.9.9.9
    - name: cache
      parameters: 30
    - name: loop
    - name: reload
    - name: loadbalance
    - name: autopath
      parameters: "@kubernetes"

Note that if i remove the kubernetes plugin, its work without issue (without internal resolving ofc). Also i've tried to increase verbosity with debug and log plugin without any issue (i saw afterwards that the kubernetes plugin contains almost no debug log).

Here are the log from the pod itself:

root@k8s-controller-1:/home/debian# kubectl logs coredns-coredns-58f47c98c8-mkpqd -n kube-system -f
[WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
.:53
[INFO] plugin/reload: Running configuration MD5 = 40faa7581f217ba0ea6e3019d2adfa0f
CoreDNS-1.8.4
linux/amd64, go1.16.4, 053c4d5
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"

I'm guessing that coredns can't fetch data from the kubernetes api but i can't understand why, i've tried to reach the api from another pod without issue:

root@my-nginx-5b56ccd65f-n5ggz:/# curl -k https://172.16.255.1    
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}

I also have checked logs for the apiserver but there is no errors or anything that links to a potential coredns request.
Some more information about my setup:

• every k8s component run on a wireguard private network (using 10.0.0.0/8 range)
• k8s pod ip are configured to use 172.16.0.0/17 and service uses 172.16.255.0/24
• cilium (v1.10.4) is configured and networking is fine across node & pods)
• I've only one worker node for my test (one cilium agent, one cilium operator, 2 nginx pod to test networking and the coredns pod)

I've looked around everywhere that i could think of without having luck, does anyone have an idea to debug this further ?
Thanks in advance

[vmarchaud]

Solved ! It was because the certificate of my kube-apiserver didn't allowed the ip of the internal kubernetes service :)

[EvoluteStudio]

so, how to fix it???????????????????????????????????????????????????????

[vmarchaud]

Well, you need to add the IP of the internal kubernetes service to the kube-apiserver certificate.

[veryrich]

please... how to add the IP of the internal kubernetes service to the kube-apiserver certificate

[ccll]

please... how to add the IP of the internal kubernetes service to the kube-apiserver certificate

Finally find the solution in k3s, if you are using other kubernetes distros, there should be similar command line options.
In k3s there is a --tls-san CLI option which allow you to add SANs to tls cert.

1. Check the service IP address of your kubernetes service.

$ kubectl get svc -A | grep kubernetes
default  kubernetes  ClusterIP  10.43.0.1  <none>  443/TCP  493d

2. Edit /etc/systemd/system/k3s.service and add the service IP to TLS-SAN.

..
ExecStart=/usr/local/bin/k3s \
                server \
                --disable traefik \
                --disable local-storage \
                --disable coredns \
                --cluster-init \
                --tls-san 10.43.0.1 \
               ...
...

3. Restart k3s and your coredns should run without problem.

$ systemctl daemon-reload
$ systemctl restart k3s

[dsyorkd]

christ on a cracker - I've ben searching for this all weekend... so simple...

[miekg]

/close

…

On Sun, 21 Nov 2021, 18:21 Valentin Marchaud, ***@***.***> wrote: Solved ! It was because the certificate of my kube-apiserver didn't allowed the ip of the internal kubernetes service :) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#4990 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACWIWYYHKJKYCD2Z2KI6PLUNETAPANCNFSM5IMQMZVQ> .

[adeel14553]

Also, i have created the k3s with sudo k3s server --flannel-backend host-gw it works great.