Openshift 4.12.9: Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"

[AnNHSVT]

Version

$ openshift-install version
[root@api ~]# openshift-install version
openshift-install 4.12.9
built from commit 15d8ffe8fa3a5ab2f36f7e0a54126c98733aaf25
release image quay.io/openshift-release-dev/ocp-release@sha256:96bf74ce789ccb22391deea98e0c5050c41b67cc17defbb38089d32226dba0b8
release architecture amd64


# Platform:
<!--
vSphere
Network: OVNKubernetes
-->

<!--
Please specify:
* IPI (automated install with `openshift-install`. If you don't know, then it's IPI)
* UPI (semi-manual installation on customised infrastructure)
-->

# What happened?
I cannot install openshift in UPI or IPI vSphere with the same errors.
[root@helper] openshift-install wait-for bootstrap-complete --dir=.
...
ERROR Bootstrap failed to complete: timed out waiting for the condition
ERROR Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane.
We have check on controller plane:
[core@master0 ~]$ ping google.com
PING google.com (142.250.66.110) 56(84) bytes of data.
64 bytes from hkg12s28-in-f14.1e100.net (142.250.66.110): icmp_seq=1 ttl=57 time=48.3 ms
64 bytes from hkg12s28-in-f14.1e100.net (142.250.66.110): icmp_seq=2 ttl=57 time=47.2 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 47.186/47.753/48.320/0.567 ms
[core@master0 ~]$ journalctl -b -f -u kubelet
-- Logs begin at Fri 2023-04-28 17:45:51 UTC. --
Apr 28 18:25:46 master0.openshiftcluster.svtech.lab kubenswrapper[2540]: E0428 18:25:46.459951    2540 kubelet.go:2396] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Apr 28 18:25:51 master0.openshiftcluster.svtech.lab kubenswrapper[2540]: E0428 18:25:51.461091    2540 kubelet.go:2396] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Apr 28 18:25:56 master0.openshiftcluster.svtech.lab kubenswrapper[2540]: E0428 18:25:56.463460    2540 kubelet.go:2396] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Apr 28 18:26:01 master0.openshiftcluster.svtech.lab kubenswrapper[2540]: E0428 18:26:01.464692    2540 kubelet.go:2396] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Apr 28 18:26:06 master0.openshiftcluster.svtech.lab kubenswrapper[2540]: E0428 18:26:06.465978    2540 kubelet.go:2396] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Apr 28 18:26:11 master0.openshiftcluster.svtech.lab kubenswrapper[2540]: E0428 18:26:11.467758    2540 kubelet.go:2396] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Apr 28 18:26:16 master0.openshiftcluster.svtech.lab kubenswrapper[2540]: E0428 18:26:16.469687    2540 kubelet.go:2396] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Apr 28 18:26:21 master0.openshiftcluster.svtech.lab kubenswrapper[2540]: E0428 18:26:21.470949    2540 kubelet.go:2396] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Apr 28 18:26:26 master0.openshiftcluster.svtech.lab kubenswrapper[2540]: E0428 18:26:26.473146    2540 kubelet.go:2396] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Apr 28 18:26:31 master0.openshiftcluster.svtech.lab kubenswrapper[2540]: E0428 18:26:31.474902    2540 kubelet.go:2396] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"

I have check https://access.redhat.com/solutions/5298091 but cannot resolve problem. Please help. 
DNS working fine in bootstrap, master can resolve domain name.
Bootstrap console: 
![image](https://user-images.githubusercontent.com/130353078/235226053-41879248-a399-4e53-8792-0237f547b093.png)

Master console: 
![image](https://user-images.githubusercontent.com/130353078/235226093-02b35eae-45a9-4a26-bf02-766320e93c6a.png)

Worker console: 
![image](https://user-images.githubusercontent.com/130353078/235226314-b848cf3e-9430-4e19-b964-08d01b93f17e.png)

[ibrassfield]

I have a similar issue when trying to install on VMWare 7.0 with UPI install method. Any solutions available to this?

[AnNHSVT]

I have resolve with re-create cluster with UPI install method by different ignition files. Still got issue in IPI install methods

[ibrassfield]

Thanks for the response... I recently got through this issue as well but what I found was I needed to remove a taint

node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

And my control-plane hosts became ready -- also had to do this for workers as well.

[adirilx]

Hi AnNHSVT, Can you please elaborate how you solved the issue, I'm facing this exact issue myself.
Best Regards.

[kaplan-michael]

This was the issue for me, already commented in #7459 (comment)

[diogocarvalhoti]

Diagnosis:

In my case, when analyzing how the network plugin was provisioned, I found that it had not been deployed due to the taint marking on the master nodes. This taint is automatically applied and removed during provisioning, and for some reason, the installation got stuck in a loop and did not execute any new tasks for a period of over 8 hours.

Taint:
node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule

Solution:

I decided to remove the taint from all the master nodes by executing the command below. After the removal, pods started to be deployed, including the one responsible for the CNI.

Command:

oc adm taint node [Master Node Name] node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule-

[ngshukun]

i untainted the master nodes and it turned ready after node rebooted, however, worker node was not able to spinned up throughout. Any advise?