Sum speed over rate-limited speed #692
Comments
|
I created a script to do the math for the interval counts and compare it to the sum. If we just calculate the non-omitted intervals, we can see the data rate is more in line with what we see per interval at 3.58Mbits/sec and the total byte count and interval byte count has a byte disparity of almost 400KB |
|
Thanks for the descriptive bug report and analysis and I apologize for not acknowledging this issue sooner. Clearly something odd is happening with our accounting. I wonder if we're somehow including the omitted bytes in the sum total or some other thing to inflate the sum? |
|
We think iperf3 is still counting bytes left over in the buffers even after iperf3 has finished measuring intervals. |
|
I'm not convinced that counting those bytes at the end is a bug (they were validly sent by the sender during the test duration, and remember the stopping conditions of the test apply at the sender side, not the receiver). However if we do count them, we need to also account for the time over which those bytes were received so the rate computation is correct. |
|
I agree. It's advantageous to keep account of all the bytes that are received and I don't see that as the bug, but rather iperf3 is not accounting for that time when those last bytes come in. Can there be another interval added to the JSON that accounts for that time and have it marked as an extra interval? |
I thought that's what we did, which is why I'm a little confused. I need to try to reproduce this bug locally and then dive into the source code. Does this happen consistently? Also does it only happen with multiple, parallel streams or do you see it with single-stream as well? |
|
It happens consistently when using the cubic algorithm and it doesn't matter whether it is a single-stream or more. It is only because we are testing lower speeds that we were able to find this issue. The extra bytes are not as noticeable on the overall SUM at higher speeds. |
|
OK. Good (?!?) news: I've been able to reproduce this problem with a couple of VMs on my laptop with a link artificially constrained by tc(8) to 256Kbps. However I've only seen this so far with multiple parallel TCP streams. I don't know at this point how significant that is, just an observation. Server is CentOS 7, client is CentOS 6, both running iperf3 compiled from the tip of It shouldn't be possible for the sender to push out 309Kbps. So I need to figure out what's going on with this. |
|
OK after playing around with this for awhile I came up with a few things. TL;DR: iperf3 is operating correctly but there are a few subtleties in interpreting the output that can be misleading.
What I think is happening here is that the sending process is able to send a large burst of data (up to the limits of the socket buffers and without regard to TCP's congestion control algorithm). At some point, the congestion control algorithms and/or packet loss force it to stop. Note that the last part of each burst is already sent as far as iperf3 is concerned (so it's counted in the interval statistics as being "sent"), but it's sitting in the socket buffers on the application side, waiting for retransmission. So it's possible for an interval to record an amount of data sent that's actually more than is possible to send over the wire, because the data hasn't actually left the sending host yet. This effect is especially visible because the short test duration (5 seconds) makes it possible to observe over a short time interval rates that aren't indicative of the longer-term rate. You don't really see this on the client (receiver) side, but the server (sender) logs show it. Here's a sender log for a recent test that I did. Note the bursty nature of traffic coming from the sender. This is visible too in the server-side output in your original bug report, although it's a little less visible (maybe because you had a larger bottleneck bandwidth?). But it shows fluctuations in the amount of data sent in each interval, both above and below the bottleneck bandwidth. Note that when I ran similar tests, but longer (say 10-30 seconds), we still see the fluctuations in data sent at the application layer per interval, but the long-term average (correctly) approaches the bottleneck bandwidth. Here's the end of a 30-second test, showing the sending and receiving bitrates roughly equal and about what we'd expect to see on a 256Kbps rate-limited link: So from what I can tell iperf3 is working correctly, but using longer tests might be needed to produce more meaningful results. (Other ways to get rid of this effect might be to use smaller sending sizes, reducing the socket buffer sizes, or using per-socket based fair queueing, although I didn't test any of those.) |
|
When the sender has a bit-rate over the rate limit it makes sense that congestion is involved, but I am more concerned with seeing that happen on the receiver end. If you look at my first client output, the receiver SUM has a bit-rate over the rate limit Why would the receiver claim to have received a rate higher than possible (in the above case it is limited to 3.75Mbits/sec)? I'd like to note that when I ran my tests, I would notice a delay in the sum results in the last interval. I usually do not see a delay in the sum results, so it makes me wonder if the receiver is still counting bytes left over in the buffer even after the last interval measurement. |
|
Acknowledged. Let me dig into this some more. The problem on the receiver end is not one that I'm seeing (maybe I was looking at the wrong part of the problem, but anywho...). I might need to figure out something else to try to reproduce that side. |
|
I spent most of a day last week trying to reproduce this bug, because the measurement accuracy (unlike some optional features) is something that's very important to us. But I haven't been able to recreate it. If somebody (anybody) can help me to see this problem first-hand and poke at it, preferably by using a couple of VMs and some limiting mechanism like tc, that'd be greatly appreciated. |
|
@bmah888 I sent you an e-mail that may be helpful |
This prevents is from reporting an incorrect number of bytes received at the end of the test, which doesn't match up with the sum of the data received during the test intervals. Log late receives if debugging mode enabled. Fixes #692. More follow-up commits coming to fix UDP and SCTP tests.
Compile-tested only at this time, run-time testing is pending. Follow-up to #692.
Don't count data for tests received after the end of a test. This prevents is from reporting an incorrect number of bytes received at the end of the test, which doesn't match up with the sum of the data received during the test intervals. Log late receives if debugging mode enabled. Fixes esnet#692.
Context
Version of iperf3: 3.2
Hardware: Between a Linux server and a Linux client
Operating system (and distribution, if any): Debian-based Linux
Bug Report
@aledesma78 was the original discoverer of this issue.
We have a device rate-limited for 3750kbps. We expected to see the final speed result to be at or below 3.75Mbits/sec
We see a final speed result above the rate-limited speed, but no non-omitted intervals reach that speed.
Rate-limited a device and run a downstream speed test.
CLIENT OUTPUT
SERVER OUTPUT
As you can see, the client's final sum reports 4.20Mbits/sec, over the rate-limit of the client. It should not be possible to achieve speeds higher than the rate-limit.
There is a noticeable delay after the final interval of the test. Is iperf3 still counting the final incoming packets after the last interval and adding them to the over total, which is then divided only by the time of the intervals? This would explain the sum being over the average of the interval speeds.
We did a wireshark capture of the incoming packets on the server side, and we noticed a lot of TCP errors (shown below). The errors were a bunch of duplicate ACKs, presumably from the burst of traffic that the cubic algorithm does while it ramps-up.
Since congestion can lead to a delay in packets, we tried the Vegas TCP congestion algorithm, to see if that made a difference.
CLIENT OUTPUT
SERVER OUTPUT
Viola! The sum is below the rate-limited speed. But there were still times when I ran iperf3 with the Vegas algorithm that I would see a sum of 3.78Mbits/sec (only slightly over rate-limited speed).
We did a wireshark capture again (shown below). This time, no TCP errors.
N/A
Enhancement Request
Sum speed seems to include more bytes than the individual interval speeds, leading to inaccurate sum speeds that are higher than they should be.
Sum speeds should include the average of the byte counts over all the intervals, any excess should be included as an additional data set.
The text was updated successfully, but these errors were encountered: