Skipping decoding of HTTP requests headers is allowed for plaintext?

[franz1981]

@nbrady-techempower
I cannot find any rule that forbid this, but I am not sure is cheating or not (here -> https://github.com/TechEmpower/FrameworkBenchmarks/wiki/Project-Information-Framework-Tests-Overview#requirements-5).
In short, if plaintext endpoint doesn't requires any of the request HTTP headers, is legit to NOT decode them at all?
This will speed up performance a lot, but if users would change the Connection header value into something different from keep-alive or just send a malformed header, it won't be detected; and that doesn't make a "production-grade" usage, that's instead, against the rule.

I assume this thing to be forbidden, but given I cannot find any specific rule, I cannot say. any idea?

[franz1981]

I see @errantmind answered few questions about http so I hope can shred some light on this

[errantmind]

Ironically, keep-alive was an unofficial extension to the http protocol in HTTP 1.0. It was a practical addition to enable connection reuse and not a part of the spec. Since then, connections have been mostly assumed to be persistent and the use of said headers is actually forbidden in HTTP 2 and HTTP 3.

As far as I know, there are no headers that are absolutely required to be sent (by the server) in all cases, although there are headers that should usually be sent, like 'Date', 'Server', 'Content-Length', and 'Content-Type', all of which are required by this benchmark.

Also, as far as I know, the server is not required to respect headers passed by the client. I'd be happy to be wrong though if anyone has some relevant info on this.

There is definitely some ambiguity around what makes a system 'production grade'. FaF is FOSS and is intentionally limited to the bare minimum of what is necessary for a very particular use case. I wouldn't recommend using FaF as a general purpose web server, however, I've used FaF in 'production' for over a year, serving millions of requests per month, but is that enough for it to be production grade? Lacking a strict definition, it is impossible to say.

It is up to the organizers of this benchmark to define the parameters of the competition and up to us, the devs, to follow them.

[franz1981]

Thanks @errantmind it helps!

Also, as far as I know, the server is not required to respect headers passed by the client. I'd be happy to be wrong though if anyone has some relevant info on this.

For this, probably @nbrady-techempower can answer (or others of Techempower team), but I would be rather surprised; I understand that it's key for techempower to still allow some degree of freedom to interpret the rules (if any, on the subject), but ignoring any client headers looks to me (naive and new on the http land) a rather shocking practice, hardly recommended in real world usage.
And still, by not doing it, it allows a web server to not being http parsing limited anymore, being just a server that decode the initial http line, detect few CRLF patterns and send back a a bunch of bytes.
I am still not sure it was in the initial purposes of this benchmark to allow that much degree of freedom, but I can be happily mistaken (and more then happy free to change Netty/Vertx/Quarkus accordly maybe, although it defeat my personal purpose of really observing what performance are in a semi-realistic use case).

[errantmind]

For sure it is up to the team to define the parameters of the competition.

When I said what you quoted above, I meant the server is not required to respect client headers with respect to any HTTP spec (AFAIK), not necessarily in this benchmark.

Side note, the benchmark is currently bottlenecked at the network level, which is why all the plaintext frameworks fluctuate in placement week to week with a little over 7 million requests per second. Parsing out all the individual headers has some performance overhead but it isn't currently the limiting factor. I would be excited to see this network limitation addressed using some of the suggestions that can be found elsewhere around here.

[franz1981]

not necessarily in this benchmark.

Thanks for the clarification!

the benchmark is currently bottlenecked at the network level, which is why all the plaintext frameworks fluctuate in placement week to week with a little over 7 million requests per second

This depends by the framework used, sadly, and not an universal property of the benchmark :"(
I can speak about the Netty case, but I've measured the same with other frameworks I cannot mention; pipelining move the cost to be equally balanced between decoding of requests/encoding of responses and I/O (in the form of both syscalls and IRQ handling, by consequence); and it's valid for many of the "nearly to the top" frameworks in plaintext.

As you rightly said "parsing" shouldn't be the costy part, but some of these frameworks (including Netty) uses internal data structures to hold parsed header names/values, often maps (which requires case-insensitive hashing of keys), with linked entries (in case of non open-addressing ones) on collisions: in short, parsing is not the bottleneck, but translating in (costy) framework objects, yep.
That's why lazy decoding header(s) would save enough cpu cycles to be able to max out the NICs and that's why I believe that having a more clear rule will make the difference.

[franz1981]

And re

I would be excited to see this network limitation addressed using some of the suggestions that can be found elsewhere around here.

Same here; we've ready some io_uring integration that we would like to see in action (that's actually a good use case for: batching syscalls among difference connections and no-syscall path while receiving data, cannot wait!)

[vietj]

I believe it would be good to have the requirement clarified

[normanmaurer]

I agree... Without clear definition / rules a benchmark becomes kind of useless imho.

[NateBrady23]

@franz1981 Yes, it was originally the intent of the Plaintext test to allow for more freedom and I think the rules we've settled on for it are probably "good enough" but I'm willing to re-evaluate. It'd be nearly impossible for us to identify all the frameworks that aren't obviously obeying rules like this because that's generally not happening in the implementation code. If you're intentionally "just a server that decode the initial http line, detect few CRLF patterns and send back a a bunch of bytes" to get in that top group of frameworks in the plaintext results, you're probably going to be marked as Stripped. We continue to have those discussions on a framework by framework basis as contributors bring it to our attention because we simply don't have the bandwidth (pun intended) to track those down.

As far as:

Without clear definition / rules a benchmark becomes kind of useless imho.

If I had a nickel for every time I heard this, I wouldn't be here. ;) The benchmarks mean something different to everyone. There are maintainers of very popular frameworks that use these benchmarks to get better. To some, they've become a core part of the development process. It may be useless to most as a "competition" and I probably agree with that. It only looks like a competition because frameworks are sorted by RPS. That's how it makes sense to display the data. We don't hand out prize money or trophies. The real winners, imo, are the ones using the benchmarks to improve their production grade frameworks.

[franz1981]

Many thanks @nbrady-techempower this is helping indeed

If you're intentionally "just a server that decode the initial http line, detect few CRLF patterns and send back a a bunch of bytes" to get in that top group of frameworks in the plaintext results, you're probably going to be marked as Stripped

Thanks, so, in this case, the very first framework I've noticed is "lazy" decoding http headers is Redkale which its approach is
"Realistic"; according to both implementation code and profiling data, plaintext won't ever trigger header parsing, because the bytecode generated application code of the plaintext endpoint never require accessing the requests headers.
But that means that headers span is barely detected (the bunch of \r\n mentioned for fun in other comments :P) for later (and never happening) decoding: see https://github.com/redkale/redkale/blob/86a7ffb642fca4a1e65a53f500b969df915f7c73/src/main/java/org/redkale/net/http/HttpRequest.java#L319.

I don't know if makes sense to open a different discussion to involve the Redkale team.

We continue to have those discussions on a framework by framework basis as contributors bring it to our attention because we simply don't have the bandwidth (pun intended) to track those down.

My personal purpose here is to learn why top performers are that fast (and kudos to the ASP.NET core team that's fully decoding everything and still a top performer) and bring netty/vertx/quarkus there: I'll share other cases while studying the rest of top performers, if can help better classify each frameworks.

The real winners, imo, are the ones using the benchmarks to improve their production grade frameworks.

Very true, indeed thanks for it, because the mentioned frameworks have benefit by this benchmark using it with such intent :)

[normanmaurer]

@franz1981 Yes, it was originally the intent of the Plaintext test to allow for more freedom and I think the rules we've settled on for it are probably "good enough" but I'm willing to re-evaluate. It'd be nearly impossible for us to identify all the frameworks that aren't obviously obeying rules like this because that's generally not happening in the implementation code. If you're intentionally "just a server that decode the initial http line, detect few CRLF patterns and send back a a bunch of bytes" to get in that top group of frameworks in the plaintext results, you're probably going to be marked as Stripped. We continue to have those discussions on a framework by framework basis as contributors bring it to our attention because we simply don't have the bandwidth (pun intended) to track those down.

As far as:

Without clear definition / rules a benchmark becomes kind of useless imho.

If I had a nickel for every time I heard this, I wouldn't be here. ;) The benchmarks mean something different to everyone. There are maintainers of very popular frameworks that use these benchmarks to get better. To some, they've become a core part of the development process. It may be useless to most as a "competition" and I probably agree with that. It only looks like a competition because frameworks are sorted by RPS. That's how it makes sense to display the data. We don't hand out prize money or trophies. The real winners, imo, are the ones using the benchmarks to improve their production grade frameworks.

I agree! That said often people look at this benchmark and pick a framework based on the numbers (which of course is not a good thing to do but it is what it is....) so it would be good if we really compare apples to apples ;)

[franz1981]

@nbrady-techempower let me know if I should open a proper issue for the RedKale case

[fakeshadow]

The difficulty would like be where we draw the line for rule of parsing. I would not throwing out any names but I've seen projects on leader board skip handling Upgrade and/or multiple value like Keep-Alive, Upgrade. Even some so called "production ready" frameworks just do headers completely wrong just for saving a couple of cycles.

IMO I'm fine with current rule set of plaintext. People can do whatever trick they want to make their numbers better but it does not matter in real world since the category has so little weight.

[franz1981]

just for saving a couple of cycles.

I think that's the key parts: is just impossible to detect automatically these (unless fuzzing requests headers and expecting specific behaviors - probably not even enforced by the HTTP protocol itself!), but is it clear there are some that are more effective then others and I think is good to have an open discussion where analysis shows (see #7984 (reply in thread)) that those are happening for a claimed "Realistic" approach (and top framework); a wrong classification won't help to compare fairly frameworks although as @nbrady-techempower said, such comparison is just a collateral effect of the RPS ordering more then the real purpose of Techempower, but still, it's there and people others then the frameworks authors can refer to it.

[errantmind]

Most (if not all) of the top Plaintext results are turning off (or not implementing) as many features as possible to score the highest. It is the nature of benchmarking efficiency that anything not included in the benchmark is dead weight.

I don't think this is a bad thing either, as long as the benchmark results are meaningful. For example, Plaintext is almost meaningful as it comes close to testing real world static site (or any pure CSR site) performance, which is a major category of all web hosting. People who host static sites do not need all the extra functionality that many of these web servers provide and may just want the fastest web server for their particular use case. To be clear, the parsing of most/all headers on the server is not required (by Spec) but also not necessary for serving up what likely amounts to the majority of the web's public content, so why would it be strictly required to be considered 'production grade'?

So, I don't think it is useful to use a one-size-fits-all definition of web servers to gatekeep the definition of what it means for a web server to be 'production ready'. Instead the question should be whether or not the benchmark categories are meaningful representations of the dominant real-world use cases of web servers.

Also, to reiterate what I said in an above comment, I fully support the team here changing the requirements for their benchmarks, including requiring headers to be parsed, but I suggest these changes should be explicit and justified using real world use-cases, not around a vague idea of what makes a web server 'production grade/ready'.

[franz1981]

I'm not proposing that frameworks skipping key behaviours like parsing (every request) http headers to be forbidden entering in the benchmark game (in Netty/Vertx/Quarkus we disable few http validations as well), but to lower the classification of approach to something different from "Realistic" (calling out @redkale here) to "Stripped" as commented by @nbrady-techempower .
I have no rights to decide this myself, given that the difference can be blurred and I am indeed awaiting the Techempower team verdict here ^^

[errantmind]

It gets complicated when the frameworks being benchmarked get 'customized' to the point of not being representative of their 'happy paths'.

For example, Microsoft's (and others) top entry for Plaintext could very easily be considered stripped as the implementation is so far outside the bounds of how people would typically use Kestrel that it is barely recognizable, but Kestrel itself does ultimately support all 'Realistic' features. Is that ok? Note, I'm not picking on them in particular, I respect the work they have done on .Net Core and use C# professionally. However, if you examine their benchmark implementations in detail, you'll find they do not actually parse the headers beyond identifying byte boundaries for the header names and values, and then completely ignore them. They are not taking them into account at all.

[franz1981]

However, if you examine their benchmark implementations in detail, you'll find they do not actually parse the headers beyond identifying byte boundaries for the header names and values, and then completely ignore them

I've seen they decode them after parsing, but given that the benchmark itself doesn't perform any logic that requires to use them, they are not accessed; but still, parsed, decoded and stored somewhere.
I have checked their implementation as well to better understand what top performers does and it was one of the few fully performing the whole cycle around http headers, but maybe I'm missing something or you mean something different...
I was looking at https://github.com/dotnet/aspnetcore/blob/bec278eabea54f63da15e10e654bdfa4168a2479/src/Servers/Kestrel/Core/src/Internal/Http/Http1Connection.cs#L183 (and later following what happens) - do you noticed anything different?

[franz1981]

@errantmind as I have previously stated, I am pretty new to the http world, but this comment is interesting

To be clear, the parsing of most/all headers on the server is not required (by Spec) but also not necessary for serving up what likely amounts to the majority of the web's public content, so why would it be strictly required to be considered 'production grade'?

Assuming this to be correct, why sending them at this point? The benchmark could be changed to not send at all, or the risk is to benchmark how smart is a framework to avoid wasteful work according the benchmark rules (that can happen, but fall into the "Stripped" definition mentioned above, to me).
If skipping decoding and parsing all http headers is considers a realistic approach...I can just say I will learn if is the case (reading more on this subject), but I am rather surprised to hear it (and a bit shocked 😳)

[errantmind]

IMO, It is realistic for a subset of web hosting that serves static sites, but not realistic for many other types of hosting. Does a web server have to support all general purpose functionality or be considered stripped? If not, what functionality must be supported?

Web servers are only as complex as the functionality they choose to support. Some will try to facilitate every possible use case, others are content to support only specific use cases.

My point is that some care should be given in defining what realistic means, and that definition should be explicit. For example, perhaps the organizers decide that realistic includes supporting all the commonly used HTTP methods and making the headers available to the users of the framework. That would make sense, even if it excludes some frameworks that do not implement this functionality. They could state this as a part of the rules then it would be easy to identify which frameworks are realistic and which are not. Currently the definition is somewhat nebulous and open to many different interpretations.

[fakeshadow]

I don't want to go into too much detail because there has always been argument around the definition of "web framework" but these are some headers for http/1 I check in my framework:

1. Transfer-Encoding and Content-Length check to prevent potential http traffic smuggling.
2. correct parsing of Content-Length so body eof can be notified to user. I don't believe have user checking it himself is a good practice.
3. correct parsing of Connection header to properly handle connection keep-alive so it's transparent to user. The value is also used to setup websocket and/or h2c upgrade correctly but I agree upgrade handling can be handled by user manually.
4. correct parsing of Expect header to prevent malicious client send body upfront before user decide to actually consuming the body.
5. other header checks I maybe missing.

But again I don't think these should be consider rules or something. It's just my practice and others may have different opinion on handling the http spec.

[franz1981]

I haven't said yet how much I have appreciated this transparent answer...so..thanks for it!!

[remittor]

It seems to me that before starting the test, you need to check the server for the basic rules for processing HTTP requests.
For example, just now I discovered that one of the fastest server libreactor cannot respond to the "Expect: 100-continue" header.

> curl --data-binary @payload.8mb.txt -v http://127.0.0.1:8080
*   Trying 127.0.0.1:8080...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> POST / HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.74.0
> Accept: */*
> Content-Length: 8175752
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
* Done waiting for 100-continue
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: libreactor
< Date: Thu, 16 Mar 2023 12:28:20 GMT
< Content-Type: text/plain
< Content-Length: 23
<
Hello from libreactor!
* Connection #0 to host 127.0.0.1 left intact

You can see that the client curl did not wait for a response from the server.

Expected Behavior:

...
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
* Signaling end of chunked upload via terminating chunk.
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/plain
...

Note: file payload.8mb.txt contain 8MB text data
Mozilla docs: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/100

[fakeshadow]

It's bad but at least it's being honest about not supporting it. There are also frameworks on top that "pretend" to support expect header by sending HTTP/1.1 100 Continue without user consent. It makes them faster(compared to those correctly handling expect) in micro benchmark as they can always do request body pre-reading with less branches to be concerned with. So if you are checking/benchmarking with this specific rule you need extra steps.

[franz1981]

This one could be used probably, but just to classifies "Stripped" impls (that can ignore it probably? no idea)

[synopse]

Could this issue be investigated?

It pop up again as #8205

IMHO we could sum up both issue as: "If the server doesn't properly respond to Connection: Close, then it should be marked as Stripped". This is easy to check, and a fair implication of the TFB requirement #2 of a "viable implementation of HTTP".

In practice, curl -H "Connection: Close" -v localhost:8080/plaintext could be run on all endpoints which are marked as "approach": "Realistic" in their benchmark_config.json file - i.e. not only /plaintext but also all other /db /json /updates and /fortunes.

A quick source code check returned some problematic hard-coded headers (not exhaustive search - this is why we need an automated test):
https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Go/gnet/src/main.go#L29
https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Rust/faf/src/main.rs
https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Rust/ntex/src/main_plt.rs
https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/vertx/src/main/java/vertx/App.java

[synopse]

I did not try to run curl -H "Connection: Close" -v localhost:8080/plaintext with VertX. Does it close the connection?
If it does, it is fine but it may not sound good enough because the date value seems fixed in the headers, which is not allowed IIRC. It would class the implementation as "stripped": For Date, we expect that the rendered date be accurate. However, it does not need to be rendered from the system clock to a byte buffer for each request. Re-rendering once per second is an acceptable tactical optimization (and is an optimization baked into many frameworks).

[errantmind]

This just barely moves the goalpost and would achieve some filtering in the short run but nothing in the long run. If these frameworks you mention handled "Connection: close" would it stop there? I won't answer for you but I suspect it wouldn't. The people who make this benchmark can make any changes they like but I don't think a simple change like this would achieve what is desired as that would require something more systematic.

[franz1981]

@synopse It is not: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/vertx/src/main/java/vertx/App.java#L133

it update a counter every second to format date value in the right way.

[franz1981]

@errantmind Agree on that: redkale does check for Connection's Close presence to decide if lazy decode HTTP params or the user code make use of it: if not, it's not decoded; that's why they are that fast. If we add the rule to check for Close they can still decide to partially decode it, performing a fast SWAR check for Connection in the headers - in short: unless there's a TCK to run, if you make the rule, there's always a way to optimize things around that.

[fakeshadow]

My stance on this matter is like my previous reply and there can be no hard rule on the definition of "real world". That said I believe there can be an alternative way of handling it. Which is we can encourage benchmark maintainer marking bench as "stripped" if they themselves consider it non realistic. And at the same time we make sure viewers be aware of it and when they find something they consider "bullshit benchmark toy" they can filter out it's result.