Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

EVPN RT-5 Gateway IP won't get used as overlay index #15844

Open
2 tasks done
v435 opened this issue Apr 25, 2024 · 0 comments
Open
2 tasks done

EVPN RT-5 Gateway IP won't get used as overlay index #15844

v435 opened this issue Apr 25, 2024 · 0 comments
Labels
triage Needs further investigation

Comments

@v435
Copy link

v435 commented Apr 25, 2024

Description

Hi! For some reason even when the NVE receives an RT-5 route with Gateway IP field filled in, Gateway IP won't be used as the overlay index, even though the corresponding RT-2 route is present in the FIB. RT-5 get installed as "pure" RT-5 routes (with RMAC as the overlay index).

Topology and description

GWIP_no_OI_resolution

Systems involved

  • TS-1 (VyOS 1.4) is a tenant router that is peered to PE-1 via a BGPv4 connection. From PE-1's point of view, TS-1 resides in the vrf "Vrf1".
  • PE-1 (Debian12/FRR 10.0) is an NVE. PE-1 is peered to TS-1 via a BGPv4 session inside a vrf "Vrf1". PE-1 is EVPN-peered to PE-2.
  • PE-2 (Debian12/FRR 10.0) is an NVE which is EVPN-peered to the PE-1 router.

What the systems are doing

  1. TS-1 advertises three IP prefixes via a BGPv4 connection towards PE-1. The advertised prefixes are 192.168.51.0/24, 192.168.52.0/24, 192.168.53.0/24.
  2. PE-1 advertises these three IP prefixes towards PE-2 as RT-5 routes. The command "advertise ipv4 unicast gateway-ip" was issued on PE-1, so PE-1 fills in the Gateway IP field of the RT-5 routes with the IP address of TS-1 (192.168.100.1).
  3. PE-1 also advertises an RT-2 route for TS-1's IP address (192.168.100.1 on eth1).
  4. PE-2 receives the RT-5 routes and TS-1's RT-2 route from PE-1 and installs them in the FIB.
  5. PE-2 has the command "enable-resolve-overlay-index" configured under the underlay BGP EVPN configuration section.

This is how Wireshark sees one of the RT-5 routes (note that the Gateway IP is non-zero). This was captured on PE-2's e1 interface (facing PE-1).

EVPN NLRI: IP Prefix route
    Route Type: IP Prefix route (5)
    Length: 34
    Route Distinguisher: 00010a8100010001 (10.129.0.1:1)
    ESI: 00:00:00:00:00:00:00:00:00:00
    Ethernet Tag ID: 0
    IP prefix length: 24
    IPv4 address: 192.168.51.0
    IPv4 Gateway address: 192.168.100.1
    VNI: 1000

Version

PE2# show version
FRRouting 10.0 (PE2) on Linux(6.1.0-18-amd64).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
    '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--sbindir=/usr/lib/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--enable-rpki' '--disable-scripting' '--enable-pim6d' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'

How to reproduce

TS-1 Config:

interfaces {
    dummy dum51 {
        address 192.168.51.1/24
    }
    dummy dum52 {
        address 192.168.52.1/24
    }
    dummy dum53 {
        address 192.168.53.1/24
    }
    ethernet eth1 {
        address 192.168.100.1/24
}
protocols {
    bgp {
        address-family {
            ipv4-unicast {
                network 192.168.51.0/24 {
                }
                network 192.168.52.0/24 {
                }
                network 192.168.53.0/24 {
                }
            }
        }
        neighbor 192.168.100.254 {
            address-family {
                ipv4-unicast {
                }
            }
            remote-as 65001
        }
        system-as 65010
    }
    static {
        route 0.0.0.0/0 {
            next-hop 192.168.100.254 {
            }
        }
    }
}

system {
    host-name TS-1
}

PE-1

PE-1 FRR Config:

!
frr version 10.0
frr defaults traditional
hostname PE1
service integrated-vtysh-config
!
vrf Vrf1
 vni 1000
exit-vrf
!
router bgp 65001
 bgp router-id 10.129.0.1
 no bgp ebgp-requires-policy
 neighbor 10.129.11.2 remote-as 65002
 !
 address-family ipv4 unicast
  network 10.129.0.1/32
  network 10.129.11.0/24
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor 10.129.11.2 activate
  advertise-all-vni
  vni 1100
   rd 10.129.0.1:1
   route-target import 1:1
   route-target export 1:1
  exit-vni
 exit-address-family
exit
!
router bgp 65001 vrf Vrf1
 no bgp ebgp-requires-policy
 neighbor 192.168.100.1 remote-as 65010
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  advertise ipv4 unicast gateway-ip
  rd 10.129.0.1:1
  route-target import 1:1
  route-target export 1:1
 exit-address-family
exit
!
end

PE-1 OS Config

#!/bin/sh
 
##############
### sysctl ###
##############
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv4.conf.ens3.forwarding=0
sysctl -w net.ipv6.conf.all.forwarding=1
 
sysctl -w net.ipv4.conf.default.rp_filter=0
sysctl -w net.ipv4.conf.all.rp_filter=0
 
###############
### OS Conf ###
###############
hostnamectl set-hostname PE1
 
###########################
### Physical Interfaces ###
###########################
ip link add dummy0 type dummy
ip address add 10.129.0.1/32 dev dummy0
ip link set dummy0 up
 
ip link set dev ens4 name e1
ip address add 10.129.11.1/24 dev e1
ip link set dev e1 up
 
ip link set dev ens9 name e6
 
###########
### VRF ###
###########
ip link add Vrf1 type vrf table 1000
ip link set Vrf1 up
 
#############
### VXLAN ###
#############
ip link add br0 type bridge
ip link set br0 master Vrf1
ip link set br0 addr 54:aa:aa:aa:aa:aa
 
nft 'add table bridge EBTABLES'
nft 'add chain bridge EBTABLES forward { type filter hook forward priority 0; }'
nft 'add rule bridge EBTABLES forward obrname "br0" ether daddr 54:aa:aa:aa:aa:aa drop'
 
ip link set br0 up
 
### e6
ip link set e6 master br0
ip link set e6 up
 
### VNI 1100
nft 'add rule bridge EBTABLES forward obrname "br0" arp daddr ip 192.168.100.254 drop'

ip address add 192.168.100.254/24 dev br0
ip link set br0 up
sysctl -w net.ipv4.conf.br0.arp_accept=1
 
ip link add vni1100 type vxlan local 10.129.0.1 dstport 4789 id 1100 nolearning
ip link set vni1100 master br0 addrgenmode none
ip link set vni1100 type bridge_slave neigh_suppress on learning off
ip link set vni1100 up
 
### L3 VNI
ip link add br1 type bridge
ip link set br1 addr 54:00:02:00:01:00
ip link set br1 master Vrf1
ip link set br1 up

ip link add vni1000 type vxlan local 10.129.0.1 dstport 4789 id 1000 nolearning
ip link set vni1000 master br1 addrgenmode none
ip link set vni1000 type bridge_slave neigh_suppress on learning off
ip link set vni1000 up

PE-2

PE-2 FRR Config

!
frr version 10.0
frr defaults traditional
hostname PE2
log syslog informational
service integrated-vtysh-config
!
vrf Vrf1
 vni 1000
exit-vrf
!
router bgp 65002
 bgp router-id 10.129.0.2
 no bgp ebgp-requires-policy
 no bgp enforce-first-as
 neighbor 10.129.11.1 remote-as 65001
 !
 address-family ipv4 unicast
  network 10.129.0.2/32
  network 10.129.12.0/24
 exit-address-family
 !
 address-family l2vpn evpn
  neighbor 10.129.11.1 activate
  advertise-all-vni
  vni 1100
   rd 10.129.0.2:1
   route-target import 1:1
   route-target export 1:1
  exit-vni
  enable-resolve-overlay-index
 exit-address-family
exit
!
router bgp 65002 vrf Vrf1
 no bgp enforce-first-as
 !
 address-family ipv4 unicast
  redistribute connected
 exit-address-family
 !
 address-family l2vpn evpn
  advertise ipv4 unicast
  rd 10.129.0.2:1
  route-target import 1:1
  route-target export 1:1
 exit-address-family
exit
!
end

PE-2 OS Config:

#!/bin/sh
 
##############
### sysctl ###
##############
sysctl -w net.ipv4.conf.all.forwarding=1
sysctl -w net.ipv4.conf.ens3.forwarding=0
sysctl -w net.ipv6.conf.all.forwarding=1
 
sysctl -w net.ipv4.conf.default.rp_filter=0
sysctl -w net.ipv4.conf.all.rp_filter=0
 
###############
### OS Conf ###
###############
hostnamectl set-hostname PE2
 
###########################
### Physical Interfaces ###
###########################
ip link add dummy0 type dummy
ip address add 10.129.0.2/32 dev dummy0
ip link set dummy0 up
 

ip link set dev ens4 name e1
ip address add 10.129.11.2/24 dev e1
ip link set dev e1 up
 

ip link set dev ens9 name e6
 
###########
### VRF ###
###########
ip link add Vrf1 type vrf table 1000
ip link set Vrf1 up
 
#############
### VXLAN ###
#############
ip link add br0 type bridge
ip link set br0 master Vrf1
 
nft 'add table bridge EBTABLES'
nft 'add chain bridge EBTABLES forward { type filter hook forward priority 0; }'
nft 'add rule bridge EBTABLES forward obrname "br0" ether daddr 54:aa:aa:aa:aa:aa drop'
 
### e6
ip link set e6 master br0
ip link set e6 up
 
### VLAN 100 / VNI 100
nft 'add rule bridge EBTABLES forward obrname "br0" arp daddr ip 192.168.100.254 drop'
 
ip address add 192.168.100.254/24 dev br0
ip link set br0 up
sysctl -w net.ipv4.conf.br0.arp_accept=1
 
ip link add vni1100 type vxlan local 10.129.0.2 dstport 4789 id 1100 nolearning
ip link set vni1100 master br0 addrgenmode none
ip link set vni1100 type bridge_slave neigh_suppress on learning off
ip link set vni1100 up
 
### L3 VNI
ip link add br1 type bridge
ip link set br1 addr 54:00:02:00:02:00
ip link set br1 master Vrf1
ip link set br1 up

ip link add vni1000 type vxlan local 10.129.0.2 dstport 4789 id 1000 nolearning
ip link set vni1000 master br1 addrgenmode none
ip link set vni1000 type bridge_slave neigh_suppress on learning off
ip link set vni1000 up

Expected behavior

The RT-5 routes get installed with TS-1's IP address (192.168.100.1) as the next-hop. The next-hop is resolved recursively through the RT-2 route for the TS-1 address.

Actual behavior

The RT-5 routes get installed as "pure" RT-5 routes, with the next-hop set to PE-1's VTEP address.

PE-2 FIB:

PE2# show ip route vrf Vrf1
Codes: K - kernel route, C - connected, L - local, S - static,
       R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric, t - Table-Direct,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF Vrf1:
B>* 192.168.51.0/24 [20/0] via 10.129.0.1, br1 onlink, weight 1, 00:20:15
B>* 192.168.52.0/24 [20/0] via 10.129.0.1, br1 onlink, weight 1, 00:20:15
B>* 192.168.53.0/24 [20/0] via 10.129.0.1, br1 onlink, weight 1, 00:20:15
C>* 192.168.100.0/24 is directly connected, br0, 00:24:46
B>* 192.168.100.1/32 [20/0] via 10.129.0.1, br1 onlink, weight 1, 00:20:15
L>* 192.168.100.254/32 is directly connected, br0, 00:24:46

Additional context

Looking at the source code it seems to me that the issue could be related to where FRR looks for the resolved next-hop. It seems to look it up in the table which you can see by issuing the "show bgp nexthop" command which contains next-hops of the underlay BGP session. But it's hard for me to say for certain.

Checklist

  • I have searched the open issues for this bug.
  • I have not included sensitive information in this report.
@v435 v435 added the triage Needs further investigation label Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
triage Needs further investigation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@v435