kube-dns ContainerCreating /run/flannel/subnet.env no such file

[kolisko]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

BUG REPORT

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):
kube-dns kubernetes setupnetworkerror flannel subnet.env no such file

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration:

VMWare Fusion for Mac

• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

• Kernel (e.g. uname -a):

Linux ubuntu-master 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

• Others:

What happened:

kube-system   kube-dns-654381707-w4mpg                0/3       ContainerCreating   0          2m

FirstSeen     LastSeen        Count   From                    SubobjectPath   Type            Reason          Message
  ---------     --------        -----   ----                    -------------   --------        ------          -------
  3m            3m              1       {default-scheduler }                    Normal          Scheduled       Successfully assigned kube-dns-654381707-w4mpg to ubuntu-master
  2m            1s              177     {kubelet ubuntu-master}                 Warning         FailedSync      Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-654381707-w4mpg_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-654381707-w4mpg_kube-system(8ffe3172-a739-11e6-871f-000c2912631c)\" using network plugins \"cni\": open /run/flannel/subnet.env: no such file or directory; Skipping pod"

What you expected to happen:

kube-dns Running

How to reproduce it (as minimally and precisely as possible):

root@ubuntu-master:~# kubeadm init
Running pre-flight checks
<master/tokens> generated token: "247a8e.b7c8c1a7685bf204"
<master/pki> generated Certificate Authority key and certificate:
Issuer: CN=kubernetes | Subject: CN=kubernetes | CA: true
Not before: 2016-11-10 11:40:21 +0000 UTC Not After: 2026-11-08 11:40:21 +0000 UTC
Public: /etc/kubernetes/pki/ca-pub.pem
Private: /etc/kubernetes/pki/ca-key.pem
Cert: /etc/kubernetes/pki/ca.pem
<master/pki> generated API Server key and certificate:
Issuer: CN=kubernetes | Subject: CN=kube-apiserver | CA: false
Not before: 2016-11-10 11:40:21 +0000 UTC Not After: 2017-11-10 11:40:21 +0000 UTC
Alternate Names: [172.20.10.4 10.96.0.1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local]
Public: /etc/kubernetes/pki/apiserver-pub.pem
Private: /etc/kubernetes/pki/apiserver-key.pem
Cert: /etc/kubernetes/pki/apiserver.pem
<master/pki> generated Service Account Signing keys:
Public: /etc/kubernetes/pki/sa-pub.pem
Private: /etc/kubernetes/pki/sa-key.pem
<master/pki> created keys and certificates in "/etc/kubernetes/pki"
<util/kubeconfig> created "/etc/kubernetes/kubelet.conf"
<util/kubeconfig> created "/etc/kubernetes/admin.conf"
<master/apiclient> created API client configuration
<master/apiclient> created API client, waiting for the control plane to become ready
<master/apiclient> all control plane components are healthy after 14.053453 seconds
<master/apiclient> waiting for at least one node to register and become ready
<master/apiclient> first node is ready after 0.508561 seconds
<master/apiclient> attempting a test deployment
<master/apiclient> test deployment succeeded
<master/discovery> created essential addon: kube-discovery, waiting for it to become ready
<master/discovery> kube-discovery is ready after 1.503838 seconds
<master/addons> created essential addon: kube-proxy
<master/addons> created essential addon: kube-dns

Kubernetes master initialised successfully!

You can now join any number of machines by running the following on each node:

kubeadm join --token=247a8e.b7c8c1a7685bf204 172.20.10.4
root@ubuntu-master:~# 
root@ubuntu-master:~# 
root@ubuntu-master:~# 
root@ubuntu-master:~# kubectl get pods --all-namespaces
NAMESPACE     NAME                                    READY     STATUS              RESTARTS   AGE
kube-system   dummy-2088944543-eo1ua                  1/1       Running             0          47s
kube-system   etcd-ubuntu-master                      1/1       Running             3          51s
kube-system   kube-apiserver-ubuntu-master            1/1       Running             0          49s
kube-system   kube-controller-manager-ubuntu-master   1/1       Running             3          51s
kube-system   kube-discovery-1150918428-qmu0b         1/1       Running             0          46s
kube-system   kube-dns-654381707-mv47d                0/3       ContainerCreating   0          44s
kube-system   kube-proxy-k0k9q                        1/1       Running             0          44s
kube-system   kube-scheduler-ubuntu-master            1/1       Running             3          51s
root@ubuntu-master:~# 
root@ubuntu-master:~# 
root@ubuntu-master:~# 
root@ubuntu-master:~# kubectl apply -f https://git.io/weave-kube
daemonset "weave-net" created
root@ubuntu-master:~# 
root@ubuntu-master:~# 
root@ubuntu-master:~# 

Anything else do we need to know:

First time I ran

root@ubuntu-master:~# kubeadm init

and then downloaded kube-flannel.yml and apply

root@ubuntu-master:~# kubectl apply -f kube-flannel.yml

And then tried to join nodes. Then I reset configuration by

root@ubuntu-master:~# kubeadm reset
root@ubuntu-master:~# rm -rf .kube/

Then tried to initialize kubernetes again using weave.

[rothgar]

kubeadm uses weave by default. You should try specifying the --pod-network-cidr= to make it use flannel instead. http://kubernetes.io/docs/getting-started-guides/kubeadm/

I'm not sure if that will work but worth a shot.
I think this is a kubeadm bug. I can't label the issue but will see if @kubernetes/sig-cluster-lifecycle can help debug

[smerrill]

I can confirm that passing the --pod-network-cidr= flag to kubeadm works for me and has allowed me to spin up several clusters in AWS using Flannel as the pod network.

[oikomi]

What is the solution to this issue?

[kolisko]

I had both flannel and weave-net in cni folder. So I removed /etc/cni/net.d/flannel and kept only weave-net and it helped.

[carldanley]

I also have the exact same problem and i'm not sure why... I've done exactly what @kolisko did and i'm still seeing ContainerCreating with the following:

Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-654381707-lzh8r_kube-system" with SetupNetworkError: "Failed to setup network for pod \"kube-dns-654381707-lzh8r_kube-system(abb2894b-b175-11e6-ac63-001b78777574)\" using network plugins \"cni\": open /run/flannel/subnet.env: no such file or directory; Skipping pod"

This occurs on a fresh install for me.

[vincipdx]

I had similar issue using kubeadm and resolved it. If you are using flannel then you need to use --pod-network-cidr= flag to kubeadm as @smerrill said. I tried to use weave and still had the same issue as @carldanley. Then I realized had remnant flannel conf in /etc/cni/netd.d/ 10-flannel.conf from my previous attempt to configure kube. I removed that flannel conf and kube dns came up fine.

[vincipdx]

Also when joining the minions to the cluster. kubeadm join complains /etc/kubernetes is not empty and failed to join the minion. using --skip-preflight-checks option to join command resolves it.

[root@kube-vm2 ~]# kubeadm join --token=
Running pre-flight checks
preflight check errors:
/etc/kubernetes is not empty

[burnyd]

Has anyone fixed this issue? I tried to join the cluster using the network-cidr= and still receiving the following when I inspect the kube-dns pod

1m 0s 106 {kubelet ubuntu-kub-1} Warning FailedSync Error syncing pod, skipping: failed to "SetupNetwork" for "kube-dns-654381707-b3phy_kube-system" with SetupNetworkError: "Failed to setup network for pod "kube-dns-654381707-b3phy_kube-system(b137457b-b8b1-11e6-a06e-080027b67fcd)" using network plugins "cni": cni config unintialized; Skipping pod"

[vincipdx]

Are you using flannel and specifying --pod-network-cidr= flag to kubeadm or some other CNI add-on?

[vincipdx]

@burnyd, You must install a pod network. I had configured and setup couple of k8s clusters successfully using kubeadm with flannel and weave as pod network. Here are my suggestions and couple of troubleshooting tips. Hope this helps. If you can paste your pod logs or events that would help debug your problem further.

1. If you want to use flannel CNI, make sure you use --pod-network-cidr to kubeadm init on the master node as below
   kubeadm init --pod-network-cidr=10.244.0.0/16

2. Install a pod network
   When using CNI, you must install a pod network add-on for pods to communicate with each other. This should be done before joining minions to the cluster. The kube-dns will wait in ContainerCreating status until a pod network is installed. You can choose any one of the add-ons that meets your needs.
   For example, if using flannel download the flannel yaml file and run as follows. You just need to run it only on the master node.
   kubectl apply -f flannel.yaml

3. Make sure your network pod (s) (in this example flannel) are running. Once flannel network pod is up running, kube-dns pod will move to running state as well.
   kubectl get pods -n kube-system -o=wide
   NAME READY STATUS RESTARTS AGE IP NODE
   dummy-2088944543-iefa6 1/1 Running 0 4h
   etcd-node1 1/1 Running 0 4h
   kube-apiserver-node1 1/1 Running 0 4h
   kube-controller-manager-node1 1/1 Running 0 4h
   kube-discovery-1150918428-l7w0n 1/1 Running 0 4h
   kube-dns-654381707-mzpc5 2/3 Running 0 4h 10.244.0.2
   kube-flannel-ds-593w5 2/2 Running 0 1m
   kube-proxy-94zjz 1/1 Running 0 4h
   kube-scheduler-node1 1/1 Running 0 4h

If flannel pod is not running clean, you need to look at the pod's logs to figure out what is going on. Few things you can check while debugging flannel pod startup or pod network communication issues
a) make sure your cni plugin binaries are in place in /opt/cni/bin. You should see corresponding binaries for each CNI add-on
b) Make sure the CNI configuration file for the network add-on is in place under /etc/cni/net.d
[root@node1]# ls /etc/cni/net.d
10-flannel.conf
c) run ifconfig to check docker, flannel bridge and virtual interfaces are up

If flannel is up and running but kube-dns still spitting out "cni: cni config unintialized; Skipping pod" errors then you have to check each containers [kube-dns dnsmasq healthz] logs in kube-dns pod to see what is going on.

[RobinLe]

Thanks @vincipdx . I follow your guide and its running now.

root@k8s-master:/home/ubuntu# kubectl get pods -n kube-system -o=wide
NAME                                 READY     STATUS              RESTARTS   AGE       IP             NODE
dummy-2088944543-q3r3b               1/1       Running             0          3m        192.168.3.48   k8s-master
etcd-k8s-master                      1/1       Running             0          3m        192.168.3.48   k8s-master
kube-apiserver-k8s-master            1/1       Running             0          3m        192.168.3.48   k8s-master
kube-controller-manager-k8s-master   1/1       Running             0          3m        192.168.3.48   k8s-master
kube-discovery-1769846148-zk1qx      1/1       Running             0          3m        192.168.3.48   k8s-master
kube-dns-2924299975-471zw            0/4       ContainerCreating   0          2m        <none>         k8s-master
kube-flannel-ds-7pbm5                2/2       Running             0          2m        192.168.3.48   k8s-master
kube-proxy-z9v01                     1/1       Running             0          2m        192.168.3.48   k8s-master
kube-scheduler-k8s-master            1/1       Running             0          3m        192.168.3.48   k8s-master
root@k8s-master:/home/ubuntu# kubectl get pods -n kube-system -o=wide
NAME                                 READY     STATUS    RESTARTS   AGE       IP             NODE
dummy-2088944543-q3r3b               1/1       Running   0          5m        192.168.3.48   k8s-master
etcd-k8s-master                      1/1       Running   0          5m        192.168.3.48   k8s-master
kube-apiserver-k8s-master            1/1       Running   0          5m        192.168.3.48   k8s-master
kube-controller-manager-k8s-master   1/1       Running   0          5m        192.168.3.48   k8s-master
kube-discovery-1769846148-zk1qx      1/1       Running   0          5m        192.168.3.48   k8s-master
kube-dns-2924299975-471zw            4/4       Running   0          5m        10.244.0.2     k8s-master
kube-flannel-ds-7pbm5                2/2       Running   0          5m        192.168.3.48   k8s-master
kube-proxy-z9v01                     1/1       Running   0          5m        192.168.3.48   k8s-master
kube-scheduler-k8s-master            1/1       Running   0          5m        192.168.3.48   k8s-master

[vincipdx]

@RobinLe , Glad to hear my suggestion were helpful and you were able to get kubernetes up and running. I am working on writing an article on Kubenernestes : Troubleshooting and debugging. Hoping to finish this up soon and publish it.

[adaughterson]

--pod-network-cidr= worked for me.

Thanks!

[inge4pres]

Thank you @vincipdx, installing from the official guide this is no so clear

[alexanderilyin]

@inge4pres unfortunately it's described in docs:

https://kubernetes.io/docs/getting-started-guides/kubeadm/

If you want to use flannel as the pod network, specify --pod-network-cidr=10.244.0.0/16 if you’re using the daemonset manifest below. However, please note that this is not required for any other networks besides Flannel.

https://kubernetes.io/docs/admin/kubeadm/

If you are using flannel with this manifest you should use --pod-network-cidr=10.244.0.0/16. Most CNI based networking solutions do not require this flag.

[inge4pres]

Hi @alexanderilyin saw that later when re-reading.
Thank you for pointing out

[overip]

Hi, I am running in a problem even though I used the --pod-network-cidr option with init.
Here are the steps summarized:

1. Install k8s following https://kubernetes.io/docs/getting-started-guides/kubeadm/

2. Once I got to the 3/4 steps from above link, I downloaded the kube-flannel.yml file on master and launched it with kubectl apply -f kube-flannel.yml. When try kubectl get pods --all-namespaces I get
   NAMESPACE NAME READY STATUS RESTARTS AGE
   default kube-flannel-ds-tjbk7 2/2 Running 0 30m
   kube-system dummy-2088944543-0ww6c 1/1 Running 0 33m
   kube-system etcd-ip-xx-xx-xx-xx 1/1 Running 0 32m
   kube-system kube-apiserver-ip-xx-xx-xx-xx 1/1 Running 0 33m
   kube-system kube-controller-manager-ip-xx-xx-xx-xx 1/1 Running 0 32m
   kube-system kube-discovery-1769846148-lfktq 1/1 Running 0 33m
   kube-system kube-dns-2924299975-65h81 4/4 Running 0 32m
   kube-system kube-proxy-c85q0 1/1 Running 0 32m
   kube-system kube-scheduler-ip-xx-xx-xx-xx 1/1 Running 0 32m

3. Then as said before, on the master I run kubeadm init --pod-network-cidr=10.244.0.0/16

4. Then I try to join the master, it fails with a message like [discovery] Failed to request cluster info, will try again:.......

I can't ping between the master and minions.

Any suggestion?

[etoews]

@rothgar you mentioned that

kubeadm uses weave by default

Do you know if that's made explicit anywhere?

I wasn't able to find a reference to Weave being the default in any of:

• https://kubernetes.io/docs/getting-started-guides/kubeadm/
• https://kubernetes.io/docs/admin/addons/
• https://kubernetes.io/docs/admin/kubeadm/

I'm just wondering if I'm missing something or if it should be made explicit in one of those guides.

[rothgar]

@everett-toews The doc doesn't explicitly say which overlay it uses. I think in an ideal world the user wouldn't have to care but obviously that's not always the case.

I would agree there probably should be a note somewhere on the page that the default is weave just to let people know a little bit about what they're deploying

[luxas]

@rothgar @everett-toews Arriving late but I have to pushback on kubeadm uses weave by default.
That isn't the case or have never been. kubeadm doesn't care about networking nor set it fully up, it's out of scope.
That's left to third-party solutions like Weave and Flannel.

You can choose to deploy whatever networking solution you want as long as it satisfies CNI.

At one moment in time, Weave was used as an example of what network provider you could use. We later removed that because Kubernetes should be neutral and not recommend any specific solution.

That said, I think this issue describes a flannel bug that has been fixed in newer flannel versions, so closing.

If the problem persists, reopen in the kubeadm or flannel repo.

[tarmael]

I had this problem as well. Solution in last line.

Issue:
The issue was the tutorial I was following suggested to install flannel yml from github, which (as of writing) uses default network: 10.244.0.0/16
Where as my tutorial I was following used network: 172.30.0.0/16

This resulted in the following errors:
kubectl get pods --all-namespaces
kube-system coredns-576cbf47c7-8frqd 0/1 CrashLoopBackOff 6 7m14s
kube-system coredns-576cbf47c7-vz87b 0/1 CrashLoopBackOff 6 7m14s

kubectl describe pods coredns-576cbf47c7-8frqd
/run/flannel/subnet.env: no such file or directory

kubectl logs -n kube-system coredns-576cbf47c7-8frqd
2018/11/27 00:03:16 [FATAL] plugin/loop: Seen "HINFO IN 8845078766411623665.6668541419380549736." more than twice, loop detected

See inconsistency in installation:
kubeadm init --pod-network-cidr=172.30.0.0/16
Executing kubeadm init on network 172.30.0.0/16
...
[certificates] apiserver serving cert is signed for DNS names [kubert.local kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.122.57]

Solution:
Either reconsider the network you will use with the --pod-network-cidr= flag, or download the flannel yml and edit the network configuration in there

[sachinjambhulkar]

This has solved the issue for me, when flannel failed to come up. Refer the thread for more info: kubernetes/kubeadm#338