UE connectivity to application on AGW host -- OVS and Linux issues?

[jblakley]

Here's one that's plagued me for a couple of months. In my application , I need to route traffic from the UE to a container on the same physical server. I have tried several different methods for doing this with no success. The container is an application running OpenRTiST. I suspect something happening between OVS and Linux routing. Here is the basic network diagram. In my setup, the UE can connect to an application server on the other side of the external gateway with no issues. I can not connect to the OpenRTiST container on the Cloudlet (192.168.8.229:9098).

Basic Cloudlet setup is:

• Magma 1.8 LTE
• Containerized AGW (shows as k8s on the diagram but really it's docker-compose right now)
• System is running baremetal (no VMs).
• Eth0, Eth1, Eth2 area all separate physical interfaces.

So, in this picture a packet from the UE destined for the application on 192.168.8.229:9098 does not get there. Pings from UE to that IP address are replied to but don't actually exit OVS. TCP traffic does get out but TCP ACKs are never responded to.

Here is the Cloudlet's routing:

ip route
default via 192.168.8.1 dev eth0 proto static
10.1.0.0/24 dev mtr0 proto kernel scope link src 10.1.0.1
10.32.0.0/12 dev weave proto kernel scope link src 10.32.0.1
127.0.0.0/24 dev ipfix0 proto kernel scope link src 127.0.0.10
127.1.0.0/24 dev li_port proto kernel scope link src 127.1.0.0
169.254.0.0/16 dev mtr0 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.8.0/24 dev eth2 proto kernel scope link src 192.168.8.229
192.168.8.0/24 dev eth0 proto kernel scope link src 192.168.8.227
192.168.9.0/24 dev eth1 proto kernel scope link src 192.168.9.228
192.168.9.6 via 192.168.9.228 dev eth1 proto static
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
192.168.128.0/24 dev gtp_br0 scope link

Here are is the ping data from gtp_br0 and gtpu_sys_2152. The ping doesn't appear anywhere else. It appears that OVS is replying to the ping itself -- somehow knowing that the destination is on the local server.

$ tshark -f icmp -i gtp_br0
Capturing on 'gtp_br0'
    1 0.000000000 192.168.128.15 → 192.168.8.229 ICMP 98 Echo (ping) request  id=0x004e, seq=1/256, ttl=64
    2 0.000098401 192.168.8.229 → 192.168.128.15 ICMP 98 Echo (ping) reply    id=0x004e, seq=1/256, ttl=64 (request in 1)
$ tshark -f icmp -i gtpu_sys_2152
Capturing on 'gtpu_sys_2152'
    1 0.000000000 192.168.128.15 → 192.168.8.229 ICMP 84 Echo (ping) request  id=0x0054, seq=1/256, ttl=64
    2 0.000079921 192.168.8.229 → 192.168.128.15 ICMP 84 Echo (ping) reply    id=0x0054, seq=1/256, ttl=64 (request in 1)
    

When connecting to the application itself, the story is a little different. TCP SYN/ACK seem to be getting lost somewhere. Again, nothing appears but on these two interfaces.

$ tshark -f "port 9098" -i gtp_br0
Capturing on 'gtp_br0'
    1 0.000000000 192.168.128.15 → 192.168.8.229 TCP 74 54458 → 9098 [SYN] Seq=0 Win=65535 Len=0 MSS=1200 SACK_PERM=1 TSval=1820602449 TSecr=0 WS=512
    2 0.000189841 192.168.8.229 → 192.168.128.15 TCP 74 9098 → 54458 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1400 SACK_PERM=1 TSval=3559614106 TSecr=1820602449 WS=128
    3 1.024259407 192.168.8.229 → 192.168.128.15 TCP 74 [TCP Retransmission] 9098 → 54458 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1400 SACK_PERM=1 TSval=3559615130 TSecr=1820602449 WS=128
    4 1.026012939 192.168.128.15 → 192.168.8.229 TCP 74 [TCP Retransmission] 54458 → 9098 [SYN] Seq=0 Win=65535 Len=0 MSS=1200 SACK_PERM=1 TSval=1820603474 TSecr=0 WS=512
    5 1.026104979 192.168.8.229 → 192.168.128.15 TCP 74 [TCP Retransmission] 9098 → 54458 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1400 SACK_PERM=1 TSval=3559615131 TSecr=1820602449 WS=128
    6 3.036026062 192.168.128.15 → 192.168.8.229 TCP 74 [TCP Retransmission] 54458 → 9098 [SYN] Seq=0 Win=65535 Len=0 MSS=1200 SACK_PERM=1 TSval=1820605490 TSecr=0 WS=512
    7 3.036128103 192.168.8.229 → 192.168.128.15 TCP 74 [TCP Retransmission] 9098 → 54458 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1400 SACK_PERM=1 TSval=3559617141 TSecr=1820602449 WS=128
    8 5.052246678 192.168.8.229 → 192.168.128.15 TCP 74 [TCP Retransmission] 9098 → 54458 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1400 SACK_PERM=1 TSval=3559619158 TSecr=1820602449 WS=128
    9 7.110027489 192.168.128.15 → 192.168.8.229 TCP 74 [TCP Retransmission] 54458 → 9098 [SYN] Seq=0 Win=65535 Len=0 MSS=1200 SACK_PERM=1 TSval=1820609558 TSecr=0 WS=512
   10 7.110153090 192.168.8.229 → 192.168.128.15 TCP 74 [TCP Retransmission] 9098 → 54458 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1400 SACK_PERM=1 TSval=3559621215 TSecr=1820602449 WS=128
   11 11.168223531 192.168.8.229 → 192.168.128.15 TCP 74 [TCP Retransmission] 9098 → 54458 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1400 SACK_PERM=1 TSval=3559625274 TSecr=1820602449 WS=128
tshark -f "port 9098" -i gtpu_sys_2152
Capturing on 'gtpu_sys_2152'
    1 0.000000000 192.168.128.15 → 192.168.8.229 TCP 60 42350 → 9098 [SYN] Seq=0 Win=65535 Len=0 MSS=1200 SACK_PERM=1 TSval=1820790106 TSecr=0 WS=512
    2 0.000470923 192.168.8.229 → 192.168.128.15 TCP 60 9098 → 42350 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1400 SACK_PERM=1 TSval=3559801761 TSecr=1820790106 WS=128
    3 0.020939814 192.168.128.15 → 192.168.8.229 TCP 52 42350 → 9098 [ACK] Seq=1 Ack=1 Win=65536 Len=0 TSval=1820790130 TSecr=3559801761
    4 0.021027215 192.168.128.15 → 192.168.8.229 HTTP 304 GET / HTTP/1.1
    5 0.021088495 192.168.8.229 → 192.168.128.15 TCP 52 9098 → 42350 [ACK] Seq=1 Ack=253 Win=65024 Len=0 TSval=3559801781 TSecr=1820790133
    6 0.022737706 192.168.8.229 → 192.168.128.15 HTTP 299 HTTP/1.1 101 Switching Protocols
    7 0.023314029 192.168.8.229 → 192.168.128.15 WebSocket 71 WebSocket Binary [FIN]
    8 0.081194680 192.168.8.229 → 192.168.128.15 TCP 71 [TCP Retransmission] 9098 → 42350 [PSH, ACK] Seq=248 Ack=253 Win=65024 Len=19 TSval=3559801842 TSecr=1820790133
    9 0.254963833 192.168.128.15 → 192.168.8.229 WebSocket 304 [TCP Spurious Retransmission] WebSocket Pong
   10 0.255070513 192.168.8.229 → 192.168.128.15 TCP 64 [TCP Dup ACK 5#1] 9098 → 42350 [ACK] Seq=267 Ack=253 Win=65024 Len=0 TSval=3559802015 TSecr=1820790362 SLE=1 SRE=253
   11 0.309242140 192.168.8.229 → 192.168.128.15 TCP 318 [TCP Retransmission] 9098 → 42350 [PSH, ACK] Seq=1 Ack=253 Win=65024 Len=266 TSval=3559802070 TSecr=1820790362
   12 0.485973912 192.168.128.15 → 192.168.8.229 WebSocket 304 [TCP Spurious Retransmission] WebSocket Pong
   13 0.486042592 192.168.8.229 → 192.168.128.15 TCP 64 [TCP Dup ACK 5#2] 9098 → 42350 [ACK] Seq=267 Ack=253 Win=65024 Len=0 TSval=3559802246 TSecr=1820790594 SLE=1 SRE=253
   14 0.789216654 192.168.8.229 → 192.168.128.15 TCP 318 [TCP Retransmission] 9098 → 42350 [PSH, ACK] Seq=1 Ack=253 Win=65024 Len=266 TSval=3559802550 TSecr=1820790594
   15 0.950984250 192.168.128.15 → 192.168.8.229 WebSocket 304 [TCP Spurious Retransmission] WebSocket Pong
   16 0.951088930 192.168.8.229 → 192.168.128.15 TCP 64 [TCP Dup ACK 5#3] 9098 → 42350 [ACK] Seq=267 Ack=253 Win=65024 Len=0 TSval=3559802711 TSecr=1820791058 SLE=1 SRE=253
   17 1.717216676 192.168.8.229 → 192.168.128.15 TCP 318 [TCP Retransmission] 9098 → 42350 [PSH, ACK] Seq=1 Ack=253 Win=65024 Len=266 TSval=3559803478 TSecr=1820791058
...

I've tried a bunch of approaches:

1. Proxy through the Local GW or the Laptop
2. Terminate on the private IP of the docker container (10.17.0.3). While the pings and openrtist got through to that IP, I had similar issues (i.e., tcp retransmissions).
3. Various IP routing/iptables/flow configurations in Linux/OVS.

Let me know if you have any ideas or need more data.

[brunohcfaria]

What's the value of block_agw_local_ips in pipelined.yml?
If it's set to true, the pipelined daemon adds a drop rule for the flows with IP addresses matching the AGW local interfaces.