TFB website Homepage

[jcheron]

The first time I arrived on the tfb website, after a google search of Framework+benchmark, I stayed only a few seconds and left.
Those few seconds, where I saw a lot of barbaric names of tools I didn't know, where I misunderstood the notion of Round, and where I didn't understand what Fortunes was, were enough to convince me to go elsewhere.
Fortunately for me, I came back.

I now come to the facts.
The Techempower team has made a great contribution to Frameworks benchmarking and works on it on a continual basis. It's a considerable amount of quality work.

But the homepage of the website does not value this work, knowing that it does not offer enough readability for those who are looking for information on this subject.

Suggestion of modification

For a better access to information, I would restructure the home page to bring up:

Infos

A clear and succinct information about the project (this one is only visible if you activate the tab Introduction).
In short, the intro should allow the user to say to himself: what I came for is right here.

Direct links

Some direct entry points to relevant result selections (to be determined or refined) :

• all results
  • all frameworks with ORM (Full or Micro)
    • each language (with ORM) c#, c++, clojure ...
  • all others (Without ORM)
    • each language (without ORM) c, c# ...
      Eventually:
  • each platform
  • each server

We could even make the top performer or performers appear below each part...

To illustrate the logic, something like this (this is an unworked excerpt):

This means replacing, exclusively on the home page, the existing filters panel with direct links to certain results.

No more complete picture of results

I think we should leave it up to the user to choose what he wants to display, rather than showing him by default a table of results that is too complete or too complex, and forcing him to do manipulations to display what he's really looking for.

For the great majority of web projects, some choices are imposed (by the infrastructure, the teams, their skills), and some questions don't even arise:

• Framework or no Framework ? in this case, we have a home-made framework
• ORM or no ORM? In the case of micro-services, perhaps
• C++ or Python? not much to see, and my team already has some skills in a language.

It is more common to want to compare frameworks of the same language, using related technologies.

I think these direct entry points would also be more motivating and rewarding for teams working on frameworks :-)

[NateBrady23]

Thanks for your thoughts on this @jcheron ! cc @bhauer

[jcheron]

Thanks @nbrady-techempower
I just reread @bhauer opinion on this at Brian Hauer's personal blog.

I am in no way questioning the validity of the comparisons made; the full results must remain easily accessible.
I only think that it is one of the possible entry points, in the same way as a more classic specific comparison between fullstack frameworks of the same language, using the same database.

I'm just in favor of one-click access:

• To specific comparisons (for example java frameworks using Mysql with an ORM)
• To the general comparison that we currently know and that allows us to compare the incomparable (Raw vs ORM, C++ vs Ruby, Platform vs Framework, Mongo vs Mysql...)

And I'm convinced that this slight redesign of the home page can bring a much better access to the information that everyone wants to get by visiting TFB website.

[bhauer]

Thank you for the thoughtful input, @jcheron! It is very useful to hear how the project and its results are understood both by unfamiliar first-time visitors and by experienced contributors.

Taking your ideas as inspiration, some thoughts I'd like to run by you for discussion:

Presently, the results web site automatically directs the user to the most-recent round, and then to Fortunes on physical hardware in particular. That is, https://www.techempower.com/benchmarks/ is presently equivalent to https://www.techempower.com/benchmarks/#section=data-r18&hw=ph&test=fortune. Prior iterations of the web site landed at the introduction page.

First, I'd like your thoughts (and other contributors' thoughts) on reverting the behavior to showing the introduction when landing at https://www.techempower.com/benchmarks/. If no one has an objection in the short term, I might just make that change and see what people think.

There is a two-sentence mini-intro on the results page:

In the following tests, we have measured the performance of several web application platforms, full-stack frameworks, and micro-frameworks (collectively, "frameworks"). For more information, read the introduction, motivation, and latest environment details.

That could be rewritten or enhanced to better introduce to the project to newbies. I am open to suggestions. The reason this small paragraph is important is that even if I switch the home URL to render the introduction page, some users may arrive via a link directly to a round, perhaps with various filters enabled.

I like your idea of providing a bunch of links with pre-canned filters applied. I personally feel like those could be added under the "Results" header on the introduction page. Like you were saying, something like:

Results

View the results for Round 18:

• All results on physical hardware.
• All full-stack frameworks on physical hardware.
• All micro-frameworks on physical hardware.
• All platforms on physical hardware.
• All implementations with full ORMs on physical hardware.
• All implementations with micro ORMs on physical hardware.
• All implementations not using an ORM on physical hardware.
  ... etc ...
  ... Same for cloud ...

[jcheron]

Thank you for your open mind @bhauer !

To answer your questions:

• I believe that the mini-intro on the results page is fine, as long as that results page is no longer the home page. At the same time, English is not my native language, so I may not be in the best position to appreciate this.
• I will not switch the home page to show the introduction again, unless its content is modified, for some of the reasons below:

Displaying the results

In addition to the introduction, and to the direct access to filtered results,
I think that a site dedicated to informing about the performance of frameworks should display some results (without giving details), just a partial ranking.

It could be for example the x first of each category.

On the home page, the partial display of the results should make you want to know more.
In addition, it gives a more visual and dynamic aspect (information is updated) to the page.

If we have good products, we might as well showcase a part of them !

Limiting the categories for direct links

Even if our products are good, you shouldn't put them all on display.

If there are too many categories, we risk getting lost in it, and it takes up space:

I won't make some distinctions on the home page (since the user can then filter by himself in the results page):
Between:

• Cloud and Physical => useless since possible after (and even if the orders of magnitude are very different between Citrine and Azure, the ranking is not affected)
• Fullstack and micro framework => the nuance is sometimes fine, and the nature of the tests does not really make the difference.
• Full and Micro ORM => it's a little bit the same thing as for frameworks, both do the same work in past tests.

On the other hand, I think:

• there is a real difference between with Framework and without (this is the case of some platforms, or some languages directly requested), idem between Raw and ORM.
• that direct access by language should be allowed in each category.

Just to give you a more visual idea of what I was thinking:

Notes

For the partial display of the results:

I think a tool_shortname field should be added for each version in the framework specifications,
so that barbaric and too long names do not appear, and that the same framework is not mentioned several times, at different positions, within a category.

For displaying by language

It can be accessed by clicking on Language results (to bring up the list),
and then go directly to the desired language.

[jcheron]

To continue:

On the precision of the Informations:

Adding a tooltip on mouse-over:

About facilitating the access to information

As a user, if I choose to:

• Compare all frameworks without ORM:

  • Currently:

    1. Click on Filter panel
    2. In classification: deactivate Platform
    3. In ORM classification: deactivate Full and Micro
    4. Scroll down for seeing the validation button
    5. Click on Apply changes
       => 5 clicks!

  • With the modified homepage:

    1. Click on All frameworks without ORM
       => 1 click

• Compare java frameworks using an ORM:

  • Currently:

    1. Click on Filter panel
    2. In classification: deactivate Platform
    3. In Languages: click on Disable all
    4. Click on java
    5. In ORM classification: deactivate None and Raw
    6. Scroll down for seeing the validation button
    7. Click on Apply changes
       => 7 clicks!

  • With the modified homepage:

    1. Under the category All frameworks with ORM, click on Language results
    2. Click on java
       => 2 clicks

Imagine all the mouse batteries saved,
this slight revision is necessary to reduce global warming, just as much as the optimization of our source codes ;-)

[jcheron]

I continue my obsessive delusion

On some Ideas for Technical Specifications:

The idea is to make the categories displayed on the home page configurable, so that the Techempower team can easily modify them, without having to modify the code.

Definition of a category (displayed on the homepage)

Displays a partial ranking and allows access to a specific filtering of results

Preview sample

Sample datas

{
	"title": "All Frameworks with ORM",
	"url_params": {
		"o": "c",
		"c": "4"
	},
	"results_count": 5,
	"has_languages": true,
	"color": "#B71C1C"
}

List of categories

List of categories displayed on the home page

Sample datas

{"categories": [
			{...},{...} ...
			],
"url_params": {"test":"fortune","hw":"ph"},
"section": "data-r18"
}

Sample preview

A simple example

Complies with the existing:

Datas

{"categories": [
		{"title": "All results",
		"url_params": {	},
		"results_count": 0,
		"has_languages": false,
		"color": "#FDD735"}
		],
"url_params": {"test":"fortune","hw":"ph"},
"section": "data-r18"
}

More categories

Just an example with 4 categories, to illustrate the configurable aspect:

Datas

{"categories": [
		{
			"title": "All results",
			"url_params": {	},
			"results_count": 5,
			"has_languages": true,
			"color": "#FDD735"
		},
		{
			"title": "All Frameworks with ORM",
			"url_params": {
				"o": "c",
				"c": "4"
			},
			"results_count": 5,
			"has_languages": true,
			"color": "#B71C1C"
		},
		{ ... }
		],
"url_params": {"test":"fortune","hw":"ph"},
"section": "data-r18"
}

It's just a thought.
This way, it gives you the freedom to display what you want on the home page, and to vary the displays according to rounds or time

[jcheron]

@nbrady-techempower @bhauer
Blimey, I was satisfied with my idea...
The screens were provided only to illustrate my idea. I know that web development is your business, and I don't pretend to replace your team.
My proposal may not be timely; feel free to tell me if this is the case, I know that not all ideas are acceptable in a project.

[bhauer]

@jcheron We do very much appreciate the ideas!

I've not yet had the time to truly digest them and we have some concurrent priorities for the results web site that will command my attention when I do have the time to allocate here.

Overall, I like the idea of providing a bit more options for how to start diving into the results. And the approach you've outlined is pretty interesting!

I'm curious if anyone else has input.

[NateBrady23]

@jcheron The level of thought and detail you put into this is awesome! I wish I had the time to put that much work into a response. Like @bhauer said, there are more iterations being worked on right now to solve some problems we've been wanting to fix for some time and then there will be room to iterate on top of that.

Not to take away from anything you've done here because you make a lot of valid points, but there are some people that display the results in an unofficial capacity. It might be worth thinking about compiling a list of those unofficial results views that help people look at the data in different ways.

[jcheron]

Thank you for your answer, @nbrady-techempower .
I think it's important to start from the principle of direct links that can be easily modified and configured by your team.
This will allow you to make them evolve, depending on the feedback, or on the results of the rounds.
That way, nothing is written in stone, except the structure defining the notion of direct link, a little more elaborated than a simple hypertext link.

[joanhey]

Actual homepage

I think that is correct. You can filter in any way, and you can send an url with that filters.

Rank order

Like @bhauer #2927 (comment), I don't believe in that rank order.

Some frameworks are highly optimized only for this benchmark, but developers using it in real apps will not get that numbers. Perhaps they will have a faster application choosing a similar framework.

Saying:

In a car race don't win the faster car, win the better driver.

Adboard or tool?

This benchmark is a great tool that help to optimize stacks, configurations, code, ... for better performance in all frameworks.

Please don't conver it in an advertising board, based only in a synthetic ranking.

Priorities

To help us optimize the frameworks:

• Show result differences between runs tfb-status#7
• Add date when visualize run results tfb-status#10
• Show github changes between actual and previous run tfb-status#5
• Select hardware cloud when visualize results from Azure tfb-status#14
• Lost history #4147

And the rest of issues opened in the tfb-status https://github.com/TechEmpower/tfb-status/issues

DStats

Are collected in each run, but we don't have any way to visualize it.
We are optimizing blindly, without the cpu, memory, req per thread, ...

Why not adopt https://ajdust.github.io/tfbvis/ ?
Is it perfect ? Not, but it's useful.

My 2¢

[jcheron]

Thanks for taking the trouble to answer, @joanhey .
My suggestion only concerned TFB website Homepage, not its detailed content, as the title of the issue indicates.

Me too, I appreciate the tools provided, as well as the overview table presenting the results.
I simply suggested to allow direct access to some filtered results (an access configurable by the TechEmpower team), from the home page, rather than displaying by default the general table, which must of course remain accessible.
I thought I had sufficiently highlighted the current problem of access to information.
As I illustrated, you currently need 7 clicks to compare java Frameworks using an ORM (its just an example), who could complain about being able to get this result directly?

I don't really understand the relationship between this approach, which seeks to simplify the use (for all types of users, including us), while preserving the existing...
and the transformation into an advertising board that you talk about pejoratively.

As a French proverb says:

" Who wants to drown his dog accuses him of rabies "

[joanhey]

I only give my opinion.
They have a limited time. And I think that there are other priorities to use that time.

If you take an opinion like a personal attack or to your issue, it's wrong, because it is not so.

[misiek08]

Probably I missed something: where can I send PR for website itself?

[NateBrady23]

@misiek08 the results site is not currently open sourced