HTTP3 upload too slow

[BingoZ]

Hi, I am using aioquic version 0.9.23 to send a 20MB data based on the HTTP3 protocol. However, I found that the speed of sending data using aioquic's send_data function is relatively slow.

Upon packet capture analysis, it was observed that aioquic slices the 20MB data into segments with a maximum body size of 1200, resulting in over 60,000 bidirectional communication packets, which may be the reason for the slow transmission rate. The total time taken is around 7 seconds.

Here are my other attempts:

1. Using Python requests libaray with HTTP/1 protocol to transmit the same data packet, packet capture revealed that the maximum body length of the sent packets could reach 64,000, resulting in approximately 1200 bidirectional packets, with a total time of around 1.3 seconds.
2. Attempted to modify the maximum length of the data packets sent by aioquic, i.e., max_datagram_frame_size, but found that this configuration seems to be limited to a maximum of 1500 by PACKET_LENGTH_MAX in crypto.c，raising error with msg "aioquic._crypto.CryptoError: Invalid payload length".

Is there any way to improve the data transmission speed of HTTP3 streams?

[BingoZ]

Hi，@jlaine would it be kind of you to give any suggestions?

[rthalley]

I'm confused by the numbers in your report. Both TCP and QUIC (UDP) will be sending payloads that stay below the IP packet path MTU, which is not typically bigger than 1500 on the Internet, and should have similar numbers of packets. I measured a local HTTP transfer between an aioquic client and an aioquic server on my laptop, and it was less than 19000 packets (most of which were the 20MB payload, and a few hundred of which were acks). It also seemed plenty fast, for python anyway! I suspect there are other factors involved, but there is not enough information in your report to know. In particular, just how are you calling aioquic (besides send_data()).

[zzbingo]

I was testiing sending with aioquic example , and receiving using hypercorn, and capturing and counting packet with tcpdump.

I wonder how http1 requests performs on you laptop?

[jlaine]

max_datagram_frame_size is certainly not the setting you want, it's only used for sending datagrams encapsulated within QUIC . I don't see how changing it could have caused the error you mention.

max_datagram_size will have an impact on your throughput but you almost certainly don't want to set it as high as 1500: it refers to the UDP payload size. You need to account for IPv4/v6 + UDP overhead so that your packet still fit within the MTU.

[zzbingo]

pack streaming lasts ablout 7 seconds with aioquic and no more than 2 seconds with http1 requests, shown by tcpdump logs.

[rthalley]

You are getting huge TCP messages (almost 64KiB) because you must be running on loopback on a Linux system. On my Mac, it has datagrams bigger than the Internet MTU of 1500 in this case, but only 8KiB or so. You will not see huge packets like these over a non-loopback network.

When I run a similar test with hypercorn and a python client, I see around 0.5 seconds and 5000-odd packets total for an HTTPS TCP transfer using HTTP/2 via httpx, and around 2.0 seconds and 21000 odd packets for HTTP/3 over QUIC using aioquic. It's not yet clear to me where the time difference comes from, e.g. whether or not it is related to pacing or just the effort of handling many smaller packets. I know it isn't the raw network, as I can transfer 20MiB of data in a few hundredths of a second over loopback if I do it directly without QUIC or TLS (and also slow it down a bit so it doesn't lose packets). Also, even if bigger UDP datagrams helped (which is not clear!), it might not be worth the effort to do path MTU discovery in aioquic just for this special case. I'll see if I can find out more about where the time is spent.

[BingoZ]

Thinks for apply. But actually I was running client and server on different machines.

[rthalley]

Further investigation shows this to be a hypercorn bug related to timeout scheduling. I'll provide more details tomorrow and also open a hypercorn bug ticket.

[rthalley]

This change makes things much faster for me.

The basic problem is that Hypercorn is scheduling a new one-shot timer task every time it gets a packet. The cost of this scheduling accounts for almost the whole of the excess time. It also leads to massive waves of pointless callbacks into aioquic. This can cause further trouble in some cases as aioquic is not expecting the timer callback to be called when no timer has expired. I plan on submitting that Hypercorn patch as a PR, and also strengthening aioquic's defenses against poor timer callback invocations.

Timings on my laptop (over loopback):

Python client, Patched Hypercorn server, 20MB
TCP:   0.48 real         0.21 user         0.04 sys
QUIC:  1.03 real         0.62 user         0.08 sys

Curl client, Patched Hypercorn server, 20MB
TCP:   0.32 real         0.03 user         0.03 sys
QUIC:  0.84 real         0.07 user         0.09 sys

Python client, Nginx server, 20MB
TCP:   0.19 real         0.14 user         0.03 sys
QUIC:  0.69 real         0.47 user         0.11 sys

Curl client, Nginx server, 20MB
TCP:   0.05 real         0.01 user         0.02 sys
QUIC:  0.11 real         0.03 user         0.05 sys