Are the endpoint flags signal and subflow meant to be exclusive? or how to get MPTCP to work for both directions?

[Jimmy-Z]

Sorry if this question is too naive, I do have quite a bit of experience with Linux and networking but very new to MPTCP, I did try to search for documents but couldn't find the answer.

Network setup:
Host A: has 3 interfaces, policy routing was set and tested.
Host B: has one.

What I found working:

• Host A:

$ ip mp l
add_addr_accepted 0 subflows 3
$ ip mp e
10.0.0.1 id 1 signal
10.0.0.2 id 2 signal
10.0.0.3 id 3 signal

• Host B:

$ ip mp l
add_addr_accepted 3 subflows 3
$ ip mp e
10.0.0.100 id 1 subflow

then mptcpize iperf3 -s on Host A and mptcpize iperf3 -c 10.0.0.1 (or .2 or .3) on Host B works.

If I changed the flags on Host A from signal to subflow, then mptcpize iperf3 -s on Host B and mptcpize iperf3 -c 10.0.0.100 on Host A works.

But if both the signal and subflow flags are set, only the second scenario would work (Host A being the client), when Host A is the server, I can see in ip mp m that additional subflows were established but closed immediately, then only one of Host A's interface was utilized.

Host A is running kernel 5.15, and Host B 6.1.

Any suggestions if I want Host A to be able to act as both MPTCP server and client?

[matttbe]

Hello,

I understand that the terminology might be confusing, we should improve that somehow.

I think your configuration is OK:

• Regarding the endpoints flags:
  • signal is typically used on the server side to announce new IP address. "Typically" because it is usually the client who initiates the connection and the new subflows to new IP address advertised by the server.
  • subflow is typically used on the client side to initiate new subflows.
  • So here:
    • Host A, the server, is correctly configured with the signal flags
    • Host B, the client, has the subflow flag: OK
    • But I guess in your setup, you can have the server establishing new connection to the client without issue (no firewall, NAT, etc.)
  • It is possible to have both flags at the same time on a host but it is less common: only if the server can establish subflows but then there is a risk to hit the limits quicker if both the client and server tries to establish subflows
• Regarding the limits:
  • add_addr_accepted is typically for the client side, the number of accepted IP address notifications (ADD_ADDR) from the other peer
  • subflows is for both: incoming and outgoing subflows.
  • So here:
    • We are supposed to be OK here.
    • But just to be sure, always good to increase the limits when testing, e.g. it is possible the kernel tries to establish a new subflow on a path that is blocked and not allowing the creation of other subflows.

when Host A is the server, I can see in ip mp m that additional subflows were established but closed immediately, then only one of Host A's interface was utilized.

That's strange. The fact the subflow is established then closed is abnormal: try to increase the limits, check the packet traces (the subflow got reset?), I guess you don't have hosts in the middle that could alter the connection or block it?

Host A is running kernel 5.15, and Host B 6.1.

Do you mind trying with a more recent kernel? At least the last LTS (v6.1) on both, ideally the last stable one (v6.6/v6.5). (e.g. on Ubuntu, you can usually easily install a newer kernel: either HWE or OEM: e.g. linux-generic-hwe-22.04)

[Jimmy-Z]

Thanks for the detailed reply!

That's strange. The fact the subflow is established then closed is abnormal

It's assuring to know that my previous understanding was alright, it's just this issue made me doubt it.

try to increase the limits

So I increase the limit to 8 (apparently the max), to my disbelief (since it was previously set to 3, which it never reached), it actually worked, to a degree, now 1 additional subflow is established and it didn't close immediately. but still less ideal.

More tests show it's the subflows number on Host A that matters, and the magic number is >=7.

Coincidentally, I noticed the number of total endpoints on Host A was also 7 (sorry I simplified my case in OP), and after more tests it's indeed relevant, if I remove some endpoints to make it 4, the magic number also becomes 4.

check the packet traces (the subflow got reset?)

I haven't done this yet, since I guess that probability is minimal?

I guess you don't have hosts in the middle that could alter the connection or block it?

I simplified my case in the OP, Host A is actually behind a NAT router, I ran 3 wireguard links between A and B, so there's nothing between them, and I think my firewall configuration was alright, since when the flag was only signal it worked.

Do you mind trying with a more recent kernel?

I'll definitely try this when I get the chance, I did test directly (without wireguard, just Internet) between NAT router (which runs kernel 6.0) and Host B (6.1), the symptom is identical.

output of ip mp m:
Host A when subflows is not big enough:

[       CREATED] token=4bd2bd40 remid=0 locid=0 saddr4=<Host A Addr 0> daddr4=<Host B> sport=5201 dport=46988
[   ESTABLISHED] token=4bd2bd40 remid=0 locid=0 saddr4=<Host A Addr 0> daddr4=<Host B> sport=5201 dport=46988
[       CREATED] token=f5d62ce6 remid=0 locid=0 saddr4=<Host A Addr 0> daddr4=<Host B> sport=5201 dport=46998
[   ESTABLISHED] token=f5d62ce6 remid=0 locid=0 saddr4=<Host A Addr 0> daddr4=<Host B> sport=5201 dport=46998
[        CLOSED] token=f5d62ce6
[        CLOSED] token=4bd2bd40

Host B:

[       CREATED] token=7d5a7101 remid=0 locid=0 saddr4=<Host B> daddr4=<Host A Addr 0> sport=46988 dport=5201
[   ESTABLISHED] token=7d5a7101 remid=0 locid=0 saddr4=<Host B> daddr4=<Host A Addr 0> sport=46988 dport=5201
[     ANNOUNCED] token=7d5a7101 remid=22 daddr4=<Host A Addr 1> dport=5201
[SF_ESTABLISHED] token=7d5a7101 remid=22 locid=0 saddr4=<Host B> daddr4=<Host A Addr 1> sport=34805 dport=5201 backup=0
[       CREATED] token=1b024df6 remid=0 locid=0 saddr4=<Host B> daddr4=<Host A Addr 0> sport=46998 dport=5201
[     SF_CLOSED] token=7d5a7101 remid=22 locid=0 saddr4=<Host B> daddr4=<Host A Addr 1> sport=34805 dport=5201 backup=0
[   ESTABLISHED] token=1b024df6 remid=0 locid=0 saddr4=<Host B> daddr4=<Host A Addr 0> sport=46998 dport=5201
[     ANNOUNCED] token=1b024df6 remid=22 daddr4=<Host A Addr 1> dport=5201
[SF_ESTABLISHED] token=1b024df6 remid=22 locid=0 saddr4=<Host B> daddr4=<Host A Addr 1> sport=39961 dport=5201 backup=0
[     SF_CLOSED] token=1b024df6 remid=22 locid=0 saddr4=<Host B> daddr4=<Host A Addr 1> sport=39961 dport=5201 backup=0
[        CLOSED] token=1b024df6
[        CLOSED] token=7d5a7101

Host A when subflows is big enough:

[       CREATED] token=eb0f84a3 remid=0 locid=0 saddr4=<Host A Addr 0> daddr4=<Host B> sport=5201 dport=50696
[   ESTABLISHED] token=eb0f84a3 remid=0 locid=0 saddr4=<Host A Addr 0> daddr4=<Host B> sport=5201 dport=50696
[SF_ESTABLISHED] token=eb0f84a3 remid=0 locid=22 saddr4=<Host A Addr 1> daddr4=<Host B> sport=5201 dport=38173 backup=0
[       CREATED] token=5e788d11 remid=0 locid=0 saddr4=<Host A Addr 0> daddr4=<Host B> sport=5201 dport=50702
[   ESTABLISHED] token=5e788d11 remid=0 locid=0 saddr4=<Host A Addr 0> daddr4=<Host B> sport=5201 dport=50702
[SF_ESTABLISHED] token=5e788d11 remid=0 locid=22 saddr4=<Host A Addr 1> daddr4=<Host B> sport=5201 dport=49777 backup=0
[        CLOSED] token=5e788d11
[        CLOSED] token=eb0f84a3

Host B:

[       CREATED] token=f3a49c98 remid=0 locid=0 saddr4=<Host B> daddr4=<Host A Addr 0> sport=50696 dport=5201
[   ESTABLISHED] token=f3a49c98 remid=0 locid=0 saddr4=<Host B> daddr4=<Host A Addr 0> sport=50696 dport=5201
[     ANNOUNCED] token=f3a49c98 remid=22 daddr4=<Host A Addr 1> dport=5201
[SF_ESTABLISHED] token=f3a49c98 remid=22 locid=0 saddr4=<Host B> daddr4=<Host A Addr 1> sport=38173 dport=5201 backup=0
[       CREATED] token=f9321692 remid=0 locid=0 saddr4=<Host B> daddr4=<Host A Addr 0> sport=50702 dport=5201
[   ESTABLISHED] token=f9321692 remid=0 locid=0 saddr4=<Host B> daddr4=<Host A Addr 0> sport=50702 dport=5201
[     ANNOUNCED] token=f9321692 remid=22 daddr4=<Host A Addr 1> dport=5201
[SF_ESTABLISHED] token=f9321692 remid=22 locid=0 saddr4=<Host B> daddr4=<Host A Addr 1> sport=49777 dport=5201 backup=0
[        CLOSED] token=f9321692
[        CLOSED] token=f3a49c98

It looks like iperf3 made two connections, one for ctrl, one for test, in both connections, there's 1 add_addr announcement, and established subflow.

Another observation is which ever address I choose for the initial connection, it's always that particular address been announced, except when I chose that address for the initial connection, then another address is announced.

[Jimmy-Z]

But I guess in your setup, you can have the server establishing new connection to the client without issue (no firewall, NAT, etc.)

Yeah.

It is possible to have both flags at the same time on a host but it is less common: only if the server can establish subflows but then there is a risk to hit the limits quicker if both the client and server tries to establish subflows
e.g. it is possible the kernel tries to establish a new subflow on a path that is blocked and not allowing the creation of other subflows.

Shouldn't that be a limit on "subflows established" instead of "subflow attempts made" though?

[Jimmy-Z]

After more tinkering, I now have 4 endpoints set as signal, 4 other endpoints set as subflow, since apparently it's not allowed to have more than 8 total endpoints, Host A limits 0 8, Host B limits 8 8, then both direction works with multiple additional subflows, 4 when Host A is the server, 3 when it's the client (still a mystery for me).

[zim68]

How can we control, which address pairs are reasonable for subflow creation between client and server?

I would like to point out that it makes sense to define VLANs with their own subnets to which the relevant network interfaces that are supposed to communicate with each other belong. Corresponding routing rules in a routing table are defined for each interface.