EVPN RT-5 Gateway IP field gets cleared to zero on spine peer #15842

v435 · 2024-04-25T12:50:57Z

Description

Hi! I have encountered a bug (at least this looks like a bug to me) where a transit EVPN peer (spine) clears the Gateway IP field in an RT-5 route to zero.

The topology is as following:

TS-1 is a tenant router that is peered to PE-1 via a BGPv4 connection. From PE-1's point of view, TS-1 resides in the vrf "Vrf1".
PE-1 (Debian12/FRR 10.0) is an NVE. PE-1 is peered to TS-1 via a BGPv4 session inside a vrf "Vrf1". PE-1 is EVPN-peered to the spine P-1 router.
P-1 (Debian12/FRR 10.0) is a spine router that is EVPN peered with both PE-1 and PE-2. It only acts as an exchange between PE-1 and PE-2. It does not have any VNI configured.
PE-2 (Debian12/FRR 10.0) is an NVE which is EVPN-peered to the P-1 router.

Description of the problem:

TS-1 advertises three IP prefixes via a BGPv4 connection towards PE-1. The advertised prefixes are 192.168.51.0/24, 192.168.52.0/24, 192.168.53.0/24.
PE-1 advertises these three IP prefixes towards P-1 as RT-5 routes. The command "advertise ipv4 unicast gateway-ip" was issued on PE-1, so PE-1 fills in the Gateway IP field of the RT-5 routes with the IP address of TS-1 (192.168.100.1).
PE-1 also advertises an RT-2 route for TS-1's IP address (192.168.100.1 on eth1).
P-1 receives the RT-5 routes and the RT-2 route and advertises them towards PE-2.

However, the contents of the Gateway IP field in the RT-5 routes is lost when P-1 advertises those routes towards PE-2.

Here's how the packet looks when entering P1 (captured from P1's e1 interface, facing PE-1):

Here's how the RT-5 packet looks when leaving P1 (captured from P1's e2 interface, facing PE-2):

Version

P1# show version
FRRouting 10.0 (P1) on Linux(6.1.0-18-amd64).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
    '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--sbindir=/usr/lib/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--enable-rpki' '--disable-scripting' '--enable-pim6d' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'

How to reproduce

TS-1's config:

interfaces {
    dummy dum51 {
        address 192.168.51.1/24
    }
    dummy dum52 {
        address 192.168.52.1/24
    }
    dummy dum53 {
        address 192.168.53.1/24
    }
    ethernet eth1 {
        address 192.168.100.1/24
}
protocols {
    bgp {
        address-family {
            ipv4-unicast {
                network 192.168.51.0/24 {
                }
                network 192.168.52.0/24 {
                }
                network 192.168.53.0/24 {
                }
            }
        }
        neighbor 192.168.100.254 {
            address-family {
                ipv4-unicast {
                }
            }
            remote-as 65001
        }
        system-as 65010
    }
    static {
        route 0.0.0.0/0 {
            next-hop 192.168.100.254 {
            }
        }
    }
}

system {
    host-name TS-1
}

PE-1

PE-1 FRR config:

!
frr version 10.0
frr defaults traditional
hostname PE1
service integrated-vtysh-config
!
vrf Vrf1
 vni 1000
exit-vrf
!
router bgp 65001
 bgp router-id 10.129.0.1
 no bgp ebgp-requires-policy
 neighbor 10.129.11.1 remote-as 65100
 !
 address-family ipv4 unicast
  network 10.129.0.1/32
  network 10.129.11.0/24
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor 10.129.11.1 activate
  advertise-all-vni
  vni 1100
   rd 10.129.0.1:1
   route-target import 1:1
   route-target export 1:1
  exit-vni
 exit-address-family
exit
!
router bgp 65001 vrf Vrf1
 no bgp ebgp-requires-policy
 neighbor 192.168.100.1 remote-as 65010
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  advertise ipv4 unicast gateway-ip
  rd 10.129.0.1:1
  route-target import 1:1
  route-target export 1:1
 exit-address-family
exit
!
end

PE-1 OS Config:

#!/bin/sh
 
##############
### sysctl ###
##############
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv4.conf.ens3.forwarding=0
sysctl -w net.ipv6.conf.all.forwarding=1
 
sysctl -w net.ipv4.conf.default.rp_filter=0
sysctl -w net.ipv4.conf.all.rp_filter=0
 
###############
### OS Conf ###
###############
hostnamectl set-hostname PE1
 
###########################
### Physical Interfaces ###
###########################
ip link add dummy0 type dummy
ip address add 10.129.0.1/32 dev dummy0
ip link set dummy0 up
 
ip link set dev ens5 name e2
ip address add 10.129.11.2/24 dev e2
ip link set dev e2 up
 
ip link set dev ens9 name e6
 
###########
### VRF ###
###########
ip link add Vrf1 type vrf table 1000
ip link set Vrf1 up
 
#############
### VXLAN ###
#############
ip link add br0 type bridge
ip link set br0 master Vrf1
ip link set br0 addr 54:aa:aa:aa:aa:aa
 
nft 'add table bridge EBTABLES'
nft 'add chain bridge EBTABLES forward { type filter hook forward priority 0; }'
nft 'add rule bridge EBTABLES forward obrname "br0" ether daddr 54:aa:aa:aa:aa:aa drop'
 
ip link set br0 up
 
### e6
ip link set e6 master br0
ip link set e6 up
 
### VNI 1100
nft 'add rule bridge EBTABLES forward obrname "br0" arp daddr ip 192.168.100.254 drop'

ip address add 192.168.100.254/24 dev br0
ip link set br0 up
sysctl -w net.ipv4.conf.br0.arp_accept=1
 
ip link add vni1100 type vxlan local 10.129.0.1 dstport 4789 id 1100 nolearning
ip link set vni1100 master br0 addrgenmode none
ip link set vni1100 type bridge_slave neigh_suppress on learning off
ip link set vni1100 up
 
### L3 VNI
ip link add br1 type bridge
ip link set br1 addr 54:00:02:00:01:00
ip link set br1 master Vrf1
ip link set br1 up

ip link add vni1000 type vxlan local 10.129.0.1 dstport 4789 id 1000 nolearning
ip link set vni1000 master br1 addrgenmode none
ip link set vni1000 type bridge_slave neigh_suppress on learning off
ip link set vni1000 up

P-1

P-1 FRR Config:

!
frr version 10.0
frr defaults traditional
hostname P1
log syslog informational
service integrated-vtysh-config
!
router bgp 65100
 bgp router-id 10.129.1.1
 no bgp ebgp-requires-policy
 no bgp enforce-first-as
 neighbor EVPN peer-group
 neighbor 10.129.11.2 remote-as 65001
 neighbor 10.129.11.2 peer-group EVPN
 neighbor 10.129.12.2 remote-as 65002
 neighbor 10.129.12.2 peer-group EVPN
 !
 address-family ipv4 unicast
  network 10.129.1.1/32
  network 10.129.11.0/24
  network 10.129.12.0/24
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor EVPN activate
 exit-address-family
exit
!
end

P-1 OS Config:

#!/bin/sh

##############
### sysctl ###
##############
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv4.conf.ens3.forwarding=0
sysctl -w net.ipv6.conf.all.forwarding=1

sysctl -w net.ipv4.conf.default.rp_filter=0
sysctl -w net.ipv4.conf.all.rp_filter=0

###############
### OS Conf ###
###############
hostnamectl set-hostname P1

##################
### Interfaces ###
##################
ip link add dummy0 type dummy
ip address add 10.129.1.1/32 dev dummy0
ip link set dummy0 up

### Link to PE1
ip link set dev ens4 name e1
ip address add 10.129.11.1/24 dev e1
ip link set e1 up

### Link to PE2
ip link set dev ens5 name e2
ip address add 10.129.12.1/24 dev e2
ip link set e2 up

PE-2

PE-2 FRR Config:

!
frr version 10.0
frr defaults traditional
hostname PE2
log syslog informational
service integrated-vtysh-config
!
vrf Vrf1
 vni 1000
exit-vrf
!
router bgp 65002
 bgp router-id 10.129.0.2
 no bgp ebgp-requires-policy
 no bgp enforce-first-as
 neighbor 10.129.12.1 remote-as 65100
 !
 address-family ipv4 unicast
  network 10.129.0.2/32
  network 10.129.12.0/24
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor 10.129.12.1 activate
  advertise-all-vni
  vni 1100
   rd 10.129.0.2:1
   route-target import 1:1
   route-target export 1:1
  exit-vni
  enable-resolve-overlay-index
 exit-address-family
exit
!
router bgp 65002 vrf Vrf1
 no bgp enforce-first-as
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  advertise ipv4 unicast
  rd 10.129.0.2:1
  route-target import 1:1
  route-target export 1:1
 exit-address-family
exit
!
end

PE-2 OS Config:

#!/bin/sh
 
##############
### sysctl ###
##############
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv4.conf.ens3.forwarding=0
sysctl -w net.ipv6.conf.all.forwarding=1
 
sysctl -w net.ipv4.conf.default.rp_filter=0
sysctl -w net.ipv4.conf.all.rp_filter=0
 
###############
### OS Conf ###
###############
hostnamectl set-hostname PE2
 
###########################
### Physical Interfaces ###
###########################
ip link add dummy0 type dummy
ip address add 10.129.0.2/32 dev dummy0
ip link set dummy0 up
 
### e1
ip link set dev ens4 name e1
ip address add 10.129.12.2/24 dev e1
ip link set dev e1 up
 
ip link set dev ens9 name e6
 
###########
### VRF ###
###########
ip link add Vrf1 type vrf table 1000
ip link set Vrf1 up
 
#############
### VXLAN ###
#############
ip link add br0 type bridge
ip link set br0 master Vrf1
 
nft 'add table bridge EBTABLES'
nft 'add chain bridge EBTABLES forward { type filter hook forward priority 0; }'
nft 'add rule bridge EBTABLES forward obrname "br0" ether daddr 54:aa:aa:aa:aa:aa drop'
 
### e6
ip link set e6 master br0
ip link set e6 up
 
### VLAN 100 / VNI 100
nft 'add rule bridge EBTABLES forward obrname "br0" arp daddr ip 192.168.100.254 drop'
 
ip address add 192.168.100.254/24 dev br0
ip link set br0 up
sysctl -w net.ipv4.conf.br0.arp_accept=1
 
ip link add vni1100 type vxlan local 10.129.0.2 dstport 4789 id 1100 nolearning
ip link set vni1100 master br0 addrgenmode none
ip link set vni1100 type bridge_slave neigh_suppress on learning off
ip link set vni1100 up
 
### L3 VNI
ip link add br1 type bridge
ip link set br1 addr 54:00:02:00:02:00
ip link set br1 master Vrf1
ip link set br1 up

ip link add vni1000 type vxlan local 10.129.0.2 dstport 4789 id 1000 nolearning
ip link set vni1000 master br1 addrgenmode none
ip link set vni1000 type bridge_slave neigh_suppress on learning off
ip link set vni1000 up

Expected behavior

I expect the RT-5 routes to retain the Gateway IP field so that TS-1's IP address could be used as the next-hop for the RT-5 routes.

Actual behavior

The "Gateway IP" fields in the RT-5 routes are set to zero (which was done by P-1), so the route installed in the FIB has PE-1's loopback address as the next-hop.

PE-2 table

PE2# show ip route vrf Vrf1
Codes: K - kernel route, C - connected, L - local, S - static,
       R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric, t - Table-Direct,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF Vrf1:
B>* 192.168.51.0/24 [20/0] via 10.129.0.1, br1 onlink, weight 1, 00:39:35
B>* 192.168.52.0/24 [20/0] via 10.129.0.1, br1 onlink, weight 1, 00:39:35
B>* 192.168.53.0/24 [20/0] via 10.129.0.1, br1 onlink, weight 1, 00:39:35
C>* 192.168.100.0/24 is directly connected, br0, 01:11:35
B>* 192.168.100.1/32 [20/0] via 10.129.0.1, br1 onlink, weight 1, 00:39:35
L>* 192.168.100.254/32 is directly connected, br0, 01:11:35

Additional context

No response

Checklist

I have searched the open issues for this bug.
I have not included sensitive information in this report.

The text was updated successfully, but these errors were encountered:

v435 added the triage Needs further investigation label Apr 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EVPN RT-5 Gateway IP field gets cleared to zero on spine peer #15842

EVPN RT-5 Gateway IP field gets cleared to zero on spine peer #15842

v435 commented Apr 25, 2024 •

edited

EVPN RT-5 Gateway IP field gets cleared to zero on spine peer #15842

EVPN RT-5 Gateway IP field gets cleared to zero on spine peer #15842

Comments

v435 commented Apr 25, 2024 • edited

Description

The topology is as following:

Description of the problem:

Version

How to reproduce

TS-1's config:

PE-1

PE-1 FRR config:

PE-1 OS Config:

P-1

P-1 FRR Config:

P-1 OS Config:

PE-2

PE-2 FRR Config:

PE-2 OS Config:

Expected behavior

Actual behavior

PE-2 table

Additional context

Checklist

v435 commented Apr 25, 2024 •

edited