EVPN-VXLAN - IPv6 via SLAAC

[tobzsc]

Hello everyone,

we are still in the process of setting up our cloud environment and could use some advice.

We are using BGP+EVPN+VXLAN in our setup, quite similar to how it's described here https://vincent.bernat.ch/en/blog/2017-vxlan-bgp-evpn.
For this we're also using the modifyvxlan.sh script from here https://gist.github.com/wido/51cb9880d86f08f73766634d7f6df3f4.
IPv4 works fine but we've hit a roadblock trying to set up IPv6.

We've configured our Juniper routers to advertise a /64 in the same VNI that our VMs are in.
However our VMs do not receive an IPv6 address. We have also manually tested using rdisc6 and that fails too. Testing on a physical host not connected via our EVPN+VXLAN fabric the host receives all RAs and successfully setups the IPv6 address.

We have debugged this problem using tcpdump on the KVM host and found the following:

• The router solicitation (RS) successfully exits the VM and we can see it on the physical interface
• We can see the response - a router advertisement (RA) - on the physical interface
• We can't see the same packet on the vxlan200 or brvx-200 interface. (It also never reaches the VM)

Our conclusion was that the kernel must be dropping multicast packets somehow, but we're at a loss at figuring out why.

quick tcpdump excerpt:

13:38:59.971759 IP (tos 0x0, ttl 64, id 26030, offset 0, flags [none], proto UDP (17), length 98)
    10.255.9.204.37206 > 10.255.11.1.4789: [bad udp cksum 0x2b2a -> 0xe4c9!] VXLAN, flags [I] (0x08), vni 200
IP6 (flowlabel 0xcdb3a, hlim 255, next-header ICMPv6 (58) payload length: 8) fe80::1c00:25ff:fe00:892 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 8

13:37:58.275395 IP (tos 0x0, ttl 63, id 58024, offset 0, flags [none], proto UDP (17), length 146)
    10.255.11.1.39155 > 10.255.9.204.4789: [udp sum ok] VXLAN, flags [I] (0x0a), vni 200
IP6 (hlim 255, next-header ICMPv6 (58) payload length: 56) fe80::8ad9:8f00:c8f4:cc80 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 56
        hop limit 64, Flags [none], pref medium, router lifetime 1800s, reachable time 0ms, retrans timer 0ms

Is anyone running a similar setup and has any advice, tips or ideas for us? Please let us know. Maybe we are just missing a kernel tweak or something.

We are using CS 4.19 on AlmaLinux 9.

[wido]

We are running such setups :-) So just to check: IPv4 works just fine in the same VNI?

Can you check if you could ping the Link-Local address from any of the routers from any of the VMs?

And could you maybe post your Juniper configuration for the IRB on how you have set up the virtual gateway? There are multiple solutions to this under JunOS. And please also post the configuration for your Router Advertisement being sent.

[tobzsc]

I know you do ;)

First some background information. We are using an EVPN+MPLS setup in our Juniper backbone which is providing connectivity. We have two gateways (MX204) which in this case terminate on two spines which do EVPN-MH. This is our cloud infrastructure based on EdgeCore devices running SONiC. Here we do EVPN+VXLAN.

We have checked several things and made some observations:

• IPv4 is running without any issues
• We cannot ping link-local addresses and made the observation that multicast packets (responsible for NDP and the RAs) are arriving but somehow disappearing on our physical links (of the KVM nodes). See tcpdump above - RAs are arriving on physical interface but are never seen on our vxlan or bridge interfaces
• When using unicast RAs and sending a ping requests from the router to one of the VM it worked for a couple of minutes. We are still trying to find out what is exactly wrong here. Seems to be an NDP issue which is also related to multicast.
• Connected a physical server directly to our Juniper backbone shows no problems with v4 and v6
• We are now connecting a physical server to our EVPN+VXLAN cloud network to do the same tests here.

The main configuration parts from the Juniper gear you requested:

trehn@fra01-pod01-a11-gw01> show configuration interfaces irb unit 200
description "CloudStack Public";
bandwidth 10g;
family inet {
    rpf-check {
        mode loose;
    }
    policer {
        arp ARP-LIMIT-DEFAULT;
    }
    address 62.x.x.1/24;
}
family inet6 {
    rpf-check {
        mode loose;
    }
    nd6-max-cache 1000;
    nd6-new-hold-limit 1000;
    address 2a00:x:x:103b::1/64;
}
mac 00:00:x:x:x:00;

and

trehn@fra01-pod01-a11-gw01> show configuration protocols router-advertisement
traceoptions {
    file ipv6-nd-trace;
    flag all;
}
interface irb.200 {
    solicit-router-advertisement-unicast;
    prefix 2a00:x:x:103b::/64;
}

The configuration is the same on both gateways.

To sum it up. The multicast IPv6 packets (for NDP, RAs, etc) are arriving on our physical interfaces (on the KVM nodes) with correct VNIs, payload, etc. But they never ever arrive on the vxlan nor the bridge interface.

[wido]

Can both routers ping eachother via iPv6?

And have you also looked at this blog: https://danhearty.wordpress.com/2019/10/12/evpn-vxlan-layer-3-gateway-irb-junos/

A few things:

• I think I have seen this before... Can't remember what it was anymore
• This is not a CloudStack issue btw
• Is there no ipv6 firewall on the host dropping the packets on forward?
• Do you see any EVPN entries with MAC+IP on the host?

[tobzsc]

Finally, we found the problem which is related to VXLAN flags. When IPv6 multicast packets enter our fabric the VXLAN packet somehow gets the flags 0x0a00 instead of 0x0800, which is being ignored by the kernel and the packet is dropped. See the corresponding code fragment here: https://elixir.bootlin.com/linux/v5.14.21/source/drivers/net/vxlan.c#L1905

This seems to be a problem with SONiC itself and we will check here further.

The temporary fix is:

tc qdisc add dev ens1f0np0 clsact
tc filter add dev ens1f0np0 ingress pref 1 proto ip flower ip_proto udp dst_port 4789 action pedit munge offset 28 u8 set 0x08
tc qdisc add dev ens1f1np1 clsact
tc filter add dev ens1f1np1 ingress pref 1 proto ip flower ip_proto udp dst_port 4789 action pedit munge offset 28 u8 set 0x08