Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubeadm complains about bridge-nf-call and ip_forward if not using docker runtime #1062

Closed
sysrich opened this issue Aug 16, 2018 · 19 comments · Fixed by kubernetes/kubernetes#68624
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug.

Comments

@sysrich
Copy link

sysrich commented Aug 16, 2018

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

After booting a clean system, running kubeadm init with a CRI runtime other than docker configured produces the following pair of error messages:

	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
	[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

What you expected to happen:

Those two pre-flight checks to pass because they could be satisfied automatically.

Obviously, these are handled automatically by the starting of the docker daemon normally, but for openSUSE Kubic we are investigating using CRI-O by default, where we do not have the luxury of an overbearing daemon to meddle with such things.

Therefore this bug report is either an opportunity for kubeadm to handle things itself. I think there is a case to be that because kubeadm knows what it needs, kubeadm should modprobe br_netfilter and echo '1' > /proc/sys/net/ipv4/ip_forward automatically rather than complaining about the issues.

Alternatively, if this suggestion is not acceptable, I'd appreciate a suggestion as to how openSUSE Kubic should automatically address these issues in a way that will remain aligned with kubeadm's general expectations.

How to reproduce it (as minimally and precisely as possible):

  • Install kubeadm and cri-o
  • systemctl enable --now crio
  • Configure kubeadm to run kublet with additional args --container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=15m
  • Run kubeadm init --cri-socket /var/run/crio/crio.sock

Anything else we need to know?:

Environment:

  • Kubernetes version (use kubectl version): v1.11.2
  • Cloud provider or hardware configuration: qemu-kvm x86_64 16GB RAM 2 cores
  • OS (e.g. from /etc/os-release): opensuse-tumbleweed-kubic
  • Kernel (e.g. uname -a): 4.17.13
  • Install tools: kubeadm
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 16, 2018
@neolit123 neolit123 added the help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. label Aug 28, 2018
@outcoldman
Copy link

Does not feel like a product bug, but documentation bug (just a missing manual). Current we have only pages on how to install kubernetes with kubeadm with docker as container runtime, we should have another one for cri-o.

@Vogtinator
Copy link

IMO kubeadm should at least do the modprobe itself. Whether it's the case for ip_forward as well is debatable as that is system configuration and up to the admin.

@sysrich
Copy link
Author

sysrich commented Sep 12, 2018

@vrothberg - what do you think? Should the above 2 conditions (modprobe and sysctl) be rectified automatically by kubeadm, or do you think this is something better handled in CRI-O?

Docker does both automagically itself

And if you think CRI-O proper shouldn't take care, where do you think the dirty hack should be carried in openSUSE? in the cri-o package or in the kubeadm package? ;)

@vrothberg
Copy link

vrothberg commented Sep 12, 2018 via email

@sysrich
Copy link
Author

sysrich commented Sep 13, 2018

This issue will be handled in openSUSE with the following changes: https://build.opensuse.org/package/rdiff/devel:kubic/kubernetes?linkrev=base&rev=9

I'm planning on submitting something similar in the upstream rpm packaging promptly

priyawadhwa pushed a commit to priyawadhwa/minikube that referenced this issue Dec 7, 2018
…tables precheck error

From this issue: kubernetes/kubeadm#1062
these files need to be added to prevent this precheck error (which occurs when running any container runtime that isn't docker).

Also, save the machine config on the user's filesystem earlier so that the buildprovisioner can access it.
balopat pushed a commit to kubernetes/minikube that referenced this issue Dec 7, 2018
* Only restart docker service if container runtime is docker

Only allow the buildroot provisioner to restart docker if the container
runtime is docker. This change should fix the bug in #3424, since now
docker will not be restarted if the container runtime is containerd.

* Added files to fix FileContent--proc-sys-net-bridge-bridge-nf-call-iptables precheck error

From this issue: kubernetes/kubeadm#1062
these files need to be added to prevent this precheck error (which occurs when running any container runtime that isn't docker).

Also, save the machine config on the user's filesystem earlier so that the buildprovisioner can access it.
@itpings
Copy link

itpings commented Apr 25, 2019

Restarting Docker did the trick...Thanks you

@adobo76
Copy link

adobo76 commented Jun 13, 2019

I second adding this information to the setup page. I was seeing this error and it caused me at least 20 min of pain until I came across this thread. Thank you for sharing the solution.

@danielsand
Copy link

make sure that you follow the procedure then the error doesnt occour
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cri-o

@natemoseman
Copy link

natemoseman commented Jul 30, 2020

simply modprobe'ng things doesn't ensure they are running. If you reboot then everything breaks. It should tell you to make sure these things are enabled persistently. Which is does for the sysctl rules, but not the modules.

When following the directions on stuff like this I don't like to read into things and try to take extra steps based on hunches. If it tells me to do something, I do it. If it doesn't I don't. I wait for things to break and go back and fix them. This way I have a better idea if it's bad documentation that is the problem instead of some random thing I did when things break.

@jefflill
Copy link

I probably need to rebuild my local Hyper-V node images.

@antaloala
Copy link

antaloala commented May 12, 2022

I just fresh installed a k8s 1.24 cluster with containerd as runtime engine, using latest kubeadm tooling and following step by step the installation procedure described in latest (1.24) k8s documentation; I experienced the problem reported in this issue when raising kubeadm init and kubeadm join commands.
I solved the problem invoking (in master and worker nodes) the commands described at the beginning of this issue

modprobe br_netfilter and echo '1' > /proc/sys/net/ipv4/ip_forward

So I understand this has not been solved yet ...

P.S: It was an installation on Ubuntu-based Virtualbox VMs on a Windows10 host OS where Hyper-V has been fully disabled, so experienced issue is not related to any Hyper-V collision (not in my case).

@neolit123
Copy link
Member

See https://kubernetes.io/docs/setup/production-environment/container-runtimes/#install-and-configure-prerequisites these are still required when setting up a CR that is k8s compatible.

@antaloala
Copy link

Thanks @neolit123 (yes, it is all there .. and I did not realize on it :-(

@RicHincapie
Copy link

I just fresh installed a k8s 1.24 cluster with containerd as runtime engine, using latest kubeadm tooling and following step by step the installation procedure described in latest (1.24) k8s documentation; I experienced the problem reported in this issue when raising kubeadm init and kubeadm join commands. I solved the problem invoking (in master and worker nodes) the commands described at the beginning of this issue

modprobe br_netfilter and echo '1' > /proc/sys/net/ipv4/ip_forward

So I understand this has not been solved yet ...

P.S: It was an installation on Ubuntu-based Virtualbox VMs on a Windows10 host OS where Hyper-V has been fully disabled, so experienced issue is not related to any Hyper-V collision (not in my case).

The same pre-flight erros happened to me with kubeadm, cri-o, and kubelet on 1.22

For any one looking for a fast work around, this is the answer.

@syamprasadsiddabattula
Copy link

thank u sir.......i am happy with ur solutions sir....tQ

@Napsty
Copy link

Napsty commented Feb 14, 2023

To make this boot-resistant, add br_netfilter to the list of kernel modules to load at boot time:

echo "br_netfilter" | sudo tee -a /etc/modules

Or into a dedicated file:

echo "br_netfilter" | sudo tee /etc/modules-load.d/netfilter.conf

This assumes you are using a Linux distribution with Systemd init, systemd-modules-load.service is responsible for loading these mentioned modules.

@WillNilges
Copy link

Still getting the ip_forward error on Raspberry Pi OS (Debian 11) and Kube 1.26

        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

Anyone got a boot-resistant solution for that one?

@samnu88
Copy link

samnu88 commented Mar 20, 2024

try
modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.