Wrong ARP replies on second interface

[kvaps]

Hi we have problem, our nodes have two interfaces: eno1 and eno1d1.
In simple configuration the second one is not used and have no ip-address, but it is up:

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback ce:b8:b2:5e:ec:7a brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether c6:54:0d:d6:6c:f0 brd ff:ff:ff:ff:ff:ff
    inet 10.36.128.182/16 brd 10.36.255.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::f603:43ff:fedf:aa59/64 scope link 
       valid_lft forever preferred_lft forever
3: eno1d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 56:88:ec:cb:dc:08 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5488:ecff:fecb:dc08/64 scope link 
       valid_lft forever preferred_lft forever

When I install romana on this machine, this interface start answers on any arp's who-has requests from our network, example:

ARPING 8.8.8.8
60 bytes from 56:88:ec:cb:dc:08 (8.8.8.8): index=0 time=675.037 msec
60 bytes from 56:88:ec:cb:dc:08 (8.8.8.8): index=1 time=130.956 msec
60 bytes from 56:88:ec:cb:dc:08 (8.8.8.8): index=2 time=690.810 msec
60 bytes from 56:88:ec:cb:dc:08 (8.8.8.8): index=3 time=470.697 msec
60 bytes from 56:88:ec:cb:dc:08 (8.8.8.8): index=4 time=98.594 msec
60 bytes from 56:88:ec:cb:dc:08 (8.8.8.8): index=5 time=302.442 msec
60 bytes from 56:88:ec:cb:dc:08 (8.8.8.8): index=6 time=598.341 msec

I'll repeat that this interface does not have any IP address at all.

But this behavior continues until I shut it down:

ip link set eno1d1 down

But even if I remove any romana containers, cleanup routes and iptables, then set this interface up, it is continue flooding our network.

[chrismarino]

Hi @kvaps Guessing that an interface without an IP is confusing the Romana installer, or its possible that this is just a bug or situation Romana does not handle properely. Any chance you can put an IP on this interface to see how it behaves? @cgilmour and/or @flashvoid might have some other suggestions.

[kvaps]

Hi @chrismarino, thanks for quick reply,
I was added IP from another subnet before setup, then installed romana. The result the same, nothing changed second interface eno1d1 continue flooding my network by fake replies.

What exactly romana does with this interface, why it start doing this things?
Any idea how can I debug it?

[chrismarino]

@kvaps I don't know exactly how this can happen. Quite strange. Probably an iptable rule that's not set/working right. When you said you cleaned up iptables, did you delete table 10? That's where the Romana rules are set.

[kvaps]

@chrismarino What was done after that:

systemctl stop kubelet
docker stop `docker ps -q`
systemctl stop docker
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
ip route flush table 10
ip rule del from all lookup 10
ip link del romana-lo
ip link del dummy0

And this behavor is still there after all of this actions.

[chrismarino]

Hmmm.....way past my ability to diagnose. Need to hear back from others for more suggestions. Sorry. Send an email to info@romana.io if you want to get on Slack for lower latency replies.

[kvaps]

I think romana can add some hidden ip 0.0.0.0/0 on this interface in some different namespace.
Or somehow switch it to brodcast mode, I have no idea what is exactly romana doing with this interface and why - is second question ;)

[kvaps]

Send an email to info@romana.io...

Thanks, I'll do this!

[kvaps]

I've tested debian and ubuntu with few kernels.
The problem ocurrs only with debian installation.

# uname -a
Linux m5c3 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 GNU/Linux
# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Steps for reproduce:

kubeadm init 
kubectl apply -f https://raw.githubusercontent.com/romana/romana/master/containerize/specs/romana-kubeadm.yml