Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

windows TCP SACK slow the download speed #10343

Open
380wmda999 opened this issue Apr 30, 2024 · 9 comments
Open

windows TCP SACK slow the download speed #10343

380wmda999 opened this issue Apr 30, 2024 · 9 comments
Assignees
@kevinGC
Labels
type: enhancement New feature or request

Comments

@380wmda999
Copy link

Description

on windows 11
when I set tcpip.TCPSACKEnabled(true) I test download speed use iperf3 it just 1G/s
but when I set tcpip.TCPSACKEnabled(false) and test download speed use iperf3 is 3G/s but get panic halfway
follow is panic info:
panic: length < 0

goroutine 112 [running]:
gvisor.dev/gvisor/pkg/tcpip/stack.PacketData.CapLength({0x4290708}, 0x80000840)
D:/gocode/src/gvisor.dev/gvisor/pkg/tcpip/stack/packet_buffer.go:581 +0x5b
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*sender).splitSeg(0x432ca08, 0x428c788, 0x80000840)
D:/gocode/src/gvisor.dev/gvisor/pkg/tcpip/transport/tcp/snd.go:630 +0x12d
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*sender).maybeSendSegment(0x432ca08, 0x428c788, 0x80000840, 0x62d1c255)
D:/gocode/src/gvisor.dev/gvisor/pkg/tcpip/transport/tcp/snd.go:899 +0x197
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*sender).sendData(0x432ca08)
D:/gocode/src/gvisor.dev/gvisor/pkg/tcpip/transport/tcp/snd.go:1022 +0x1bf
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*Endpoint).sendData(0x4804508, 0x428c788)
D:/gocode/src/gvisor.dev/gvisor/pkg/tcpip/transport/tcp/connect.go:1007 +0x4d
gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*Endpoint).Write(0x4804508, {0xebfff0, 0x40f8b40}, {0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0, ...}})
D:/gocode/src/gvisor.dev/gvisor/pkg/tcpip/transport/tcp/endpoint.go:1663 +0x112
gvisor.dev/gvisor/pkg/tcpip/adapters/gonet.(*TCPConn).Write(0x4311480, {0x46ca000, 0xf000, 0xf000})
D:/gocode/src/gvisor.dev/gvisor/pkg/tcpip/adapters/gonet/gonet.go:389 +0x23a
client-cdn/tun2socks/gvisor-netstack.copyConn({0xec4d54, 0x4311480}, {0xec4b14, 0x459c3e8}, 0x0)
D:/gocode/src/client-cdn/tun2socks/gvisor-netstack/tcp.go:55 +0x138
client-cdn/tun2socks/gvisor-netstack.handleTCP(0x4311480)
D:/gocode/src/client-cdn/tun2socks/gvisor-netstack/tcp.go:42 +0x1f8
client-cdn/tun2socks/gvisor-netstack.Start.func1(0x4040390)
D:/gocode/src/client-cdn/tun2socks/gvisor-netstack/tun2socks.go:153 +0x25f
created by gvisor.dev/gvisor/pkg/tcpip/transport/tcp.(*Forwarder).HandlePacket in goroutine 303
D:/gocode/src/gvisor.dev/gvisor/pkg/tcpip/transport/tcp/forwarder.go:98 +0x2c0

adapter is wintun
I routed 192.168.3.35 to wintun gateway
ipfer3 command is: iperf3.exe -c 192.168.3.35 -R

Is this feature related to a specific bug?

No response

Do you have a specific solution in mind?

No response
@380wmda999 380wmda999 added the type: enhancement New feature or request label Apr 30, 2024
@kevinGC
Copy link
Collaborator

kevinGC commented May 10, 2024 •
edited

I'm having trouble seeing how a negative value gets passed in. That value could be negative due to casting in snd.go:

		if !seg.sequenceNumber.LessThan(end) {
			return false
		}

		available := int(seg.sequenceNumber.Size(end))
		if available == 0 {
			return false
		}

But because of the LessThan guard above, I don't see how this could happen. seg.sequenceNumber shouldn't be more than the maximum TCP window (~1GB) from end, so this should always fit neatly into an int and not overflow.

What sort of machine are you running on? Is it a 32 or 64 bit machine?

@380wmda999
Copy link
Author

380wmda999 commented May 10, 2024 •
edited

What sort of machine are you running on? Is it a 32 or 64 bit machine?

os is windows 11
I found it happened when I use go1.22.2.windows-386 compile ,
when I use go1.22.2.windows amd64 compile, it will not get panic
so it is not support for 32 bit program on windows ?

@380wmda999
Copy link
Author

There is one more question
I tested the native downlink using iperf3 on Windows, which is approximately 10G/s
But after using gvisor, even if I turn off the sack, the downlink speed is only 2G/s (if I don't turn off the sack, it's only about 1G slower). Why is this
Is it a problem with the Windows operating system or with stack performance?

@kevinGC
Copy link
Collaborator

kevinGC commented May 10, 2024

It should work on 32 bit Windows, but clearly there's a bug here (likely an overflow).

Netstack will be slower than native. And since it is rarely used on Windows, I don't think it's ever been optimized for Windows at all. That huge slowdown isn't normal (and isn't what we see on other systems), and there's also whatever tun2socks is doing that could slow things down. But native will be faster.

@380wmda999
Copy link
Author

I also suspect that overflow in 32-bit mode causes the length to become a negative number, causing panic.
But when I set TCPSACKEnabled(true), there will be no panic, but the downlink speed of the test will be very slow. So are there any optimization strategies available under Windows system?

@kevinGC
Copy link
Collaborator

kevinGC commented May 10, 2024

Unfortunately I don't think we have cycles to debug and optimize netstack on Windows, but we'd welcome the help if you're up to try.

@380wmda999
Copy link
Author

I may not have that ability because I have encountered many situations during use that I cannot understand
For example, on an Android phone, I tested the original network speed at 600M/s for downlink and 60M/s for uplink
But after using tun2socks by givsor, the downlink speed was 500m/s, but the uplink speed reached 120M/s

@kevinGC
Copy link
Collaborator

kevinGC commented May 15, 2024

Hey, could you try something for me? I have a suspicion about the slowdown from SACK. Can you apply this patch and give me the log output?

diff --git a/pkg/tcpip/transport/tcp/snd.go b/pkg/tcpip/transport/tcp/snd.go
index 5dccf81d1e...b1337faec3 100644
--- a/pkg/tcpip/transport/tcp/snd.go
+++ b/pkg/tcpip/transport/tcp/snd.go
@@ -16,6 +16,7 @@

 import (
        "fmt"
+       "log"
        "math"
        "sort"
        "time"
@@ -196,6 +197,7 @@
        // TCP options that it is including in the packets that it sends.
        // See: https://tools.ietf.org/html/rfc6691#section-2
        maxPayloadSize := int(mss) - ep.maxOptionSize()
+       log.Printf("tcp.newSender called with mss and opt size %d, %d", mss, ep.maxOptionSize())

        s := &sender{
                ep: ep,
@@ -297,6 +299,8 @@
 // attempts to retransmit the first packet above the MTU size.
 // +checklocks:s.ep.mu
 func (s *sender) updateMaxPayloadSize(mtu, count int) {
+       log.Printf("tcp.sender.updateMaxPayloadSize: mtu (%d), count (%d), s.MaxPayloadSize (%d)", mtu, count, s.MaxPayloadSize)
+
        m := mtu - header.TCPMinimumSize

        m -= s.ep.maxOptionSize()

It's possible that the endpoint is being setup with a very small MTU. In that case, enabling SACK could shrink the usable MSS enough that it affects throughput.

@kevinGC kevinGC reopened this May 15, 2024
@380wmda999
Copy link
Author

sackenable true:

2024/05/16 04:15:03 Using existing driver 0.14
2024/05/16 04:15:03 Creating adapter
2024/05/16 04:15:03 tcp.newSender called with mss and opt size 65495, 36
2024/05/16 04:15:03 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65459)
2024/05/16 04:15:04 tcp.newSender called with mss and opt size 65495, 36
2024/05/16 04:15:04 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65459)
2024/05/16 04:15:04 tcp.newSender called with mss and opt size 65495, 36
2024/05/16 04:15:04 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65459)
2024/05/16 04:15:13 tcp.newSender called with mss and opt size 65495, 36
2024/05/16 04:15:13 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65459)
2024/05/16 04:15:13 tcp.newSender called with mss and opt size 65495, 36
2024/05/16 04:15:13 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65459)

iperf out:
iperf3.exe -c 192.168.2.5 -R

Connecting to host 192.168.2.5, port 5201
Reverse mode, remote host 192.168.2.5 is sending
[  4] local 192.168.200.200 port 61449 connected to 192.168.2.5 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  68.3 MBytes   573 Mbits/sec
[  4]   1.00-2.00   sec  62.3 MBytes   523 Mbits/sec
[  4]   2.00-3.00   sec  62.8 MBytes   527 Mbits/sec
[  4]   3.00-4.00   sec  64.0 MBytes   537 Mbits/sec
[  4]   4.00-5.00   sec  65.0 MBytes   545 Mbits/sec
[  4]   5.00-6.00   sec  63.9 MBytes   536 Mbits/sec
[  4]   6.00-7.00   sec  62.8 MBytes   527 Mbits/sec
[  4]   7.00-8.00   sec  63.4 MBytes   532 Mbits/sec
[  4]   8.00-9.00   sec  62.8 MBytes   527 Mbits/sec
[  4]   9.00-10.00  sec  60.6 MBytes   509 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   648 MBytes   544 Mbits/sec                  sender
[  4]   0.00-10.00  sec   636 MBytes   534 Mbits/sec                  receiver

sackenable false:

2024/05/16 04:17:11 Using existing driver 0.14
2024/05/16 04:17:11 Creating adapter
2024/05/16 04:17:12 tcp.newSender called with mss and opt size 65495, 0
2024/05/16 04:17:12 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65495)
2024/05/16 04:17:13 tcp.newSender called with mss and opt size 65495, 0
2024/05/16 04:17:13 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65495)
2024/05/16 04:17:13 tcp.newSender called with mss and opt size 65495, 0
2024/05/16 04:17:13 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65495)
2024/05/16 04:17:18 tcp.newSender called with mss and opt size 65495, 0
2024/05/16 04:17:18 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65495)
2024/05/16 04:17:18 tcp.newSender called with mss and opt size 65495, 0
2024/05/16 04:17:18 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65495)
2024/05/16 04:17:47 tcp.newSender called with mss and opt size 65495, 0
2024/05/16 04:17:47 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65495)
2024/05/16 04:17:47 tcp.newSender called with mss and opt size 65495, 0
2024/05/16 04:17:47 tcp.sender.updateMaxPayloadSize: mtu (1480), count (0), s.MaxPayloadSize (65495)

iperf out:
iperf3.exe -c 192.168.2.5 -R

Connecting to host 192.168.2.5, port 5201
Reverse mode, remote host 192.168.2.5 is sending
[  4] local 192.168.200.200 port 61607 connected to 192.168.2.5 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   175 MBytes  1.46 Gbits/sec
[  4]   1.00-2.00   sec   169 MBytes  1.42 Gbits/sec
[  4]   2.00-3.00   sec   171 MBytes  1.43 Gbits/sec
[  4]   3.00-4.00   sec   172 MBytes  1.44 Gbits/sec
[  4]   4.00-5.00   sec   162 MBytes  1.36 Gbits/sec
[  4]   5.00-6.00   sec   167 MBytes  1.40 Gbits/sec
[  4]   6.00-7.00   sec   168 MBytes  1.41 Gbits/sec
[  4]   7.00-8.00   sec   165 MBytes  1.38 Gbits/sec
[  4]   8.00-9.00   sec   166 MBytes  1.40 Gbits/sec
[  4]   9.00-10.00  sec   164 MBytes  1.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.65 GBytes  1.42 Gbits/sec                  sender
[  4]   0.00-10.00  sec  1.64 GBytes  1.41 Gbits/sec                  receiver

ipferf raw speed out :

Connecting to host 192.168.2.5, port 5201
Reverse mode, remote host 192.168.2.5 is sending
[  4] local 192.168.2.5 port 61385 connected to 192.168.2.5 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.13 GBytes  9.73 Gbits/sec
[  4]   1.00-2.00   sec   881 MBytes  7.39 Gbits/sec
[  4]   2.00-3.00   sec  1.28 GBytes  11.0 Gbits/sec
[  4]   3.00-4.00   sec   818 MBytes  6.86 Gbits/sec
[  4]   4.00-5.01   sec  1.34 GBytes  11.4 Gbits/sec
[  4]   5.01-6.00   sec  1.07 GBytes  9.22 Gbits/sec
[  4]   6.00-7.00   sec  1022 MBytes  8.60 Gbits/sec
[  4]   7.00-8.00   sec   688 MBytes  5.77 Gbits/sec
[  4]   8.00-9.00   sec  1.09 GBytes  9.41 Gbits/sec
[  4]   9.00-10.00  sec  1.02 GBytes  8.73 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  10.3 GBytes  8.81 Gbits/sec                  sender
[  4]   0.00-10.00  sec  10.3 GBytes  8.81 Gbits/sec                  receiver

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kevinGC kevinGC

Labels
type: enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@kevinGC @380wmda999