[Java] Missing information about versions in use

[waghanza]

Hi,

In this file, we can see that http://javalite.io/activeweb use java 11

FrameworkBenchmarks/frameworks/Java/activeweb/pom.xml

Line 13 in a5aa5b3

<maven.compiler.source>11</maven.compiler.source>

However, http://actframework.org use java 8

FrameworkBenchmarks/frameworks/Java/act/pom.xml

Line 71 in a5aa5b3

<java.version>1.8</java.version>

All java framework SHOULD not use the same version ?

I'm new (again) to java world, so it have no idea of each version SHOULD be used, but seems to be 8 no ?

Regards,

[NateBrady23]

We don't require that all frameworks use the same version. That's up to the framework maintainer. Some frameworks are not ready for the newest versions. It would be nice if they all were on the latest version, but sometimes that's not possible. If one maintainer feels their framework is at a disadvantage or could benefit from a different version, it's fine for them to use that. Also, if you see that a framework could be upgraded, feel free to open a pull request to do that. Thanks!

[waghanza]

The comparison seems unfair then

[NateBrady23]

If the language version were the only thing that could affect performance, I might agree with you, but it's certainly not. And again, framework maintainers are in charge of their own updates. It's not "unfair" if some frameworks aren't yet compatible with the newest versions. These benchmarks are here to help frameworks improve, to see what others are doing to achieve better performance, etc. Not to hand out trophies. If maintainers believe they are being unfairly compared, they can remove themselves from the benchmarks. For the majority, they're here to improve, not simply to be compared.

[waghanza]

These benchmarks are here to help frameworks improve

That's not what I understood when I read https://www.techempower.com/benchmarks/#section=motivation.

BTW, I understand better this now, but it could be clarified in docs, I think

[NateBrady23]

Thanks @waghanza for pointing that out. We're actually working on a much bigger post that should better clarify our intentions.

[plokhotnyuk]

@waghanza there is also an option to run of different (J)VMs like here to greatly outperform all others FTW :)

[NateBrady23]

Right. And the options to run different servers, and different JSON serializers, and different databases and database drivers, and different all the things. This is why the test implementations are public.

Lol, poor JavaScript.

[zloster]

@waghanza

However, http://actframework.org use java 8

FrameworkBenchmarks/frameworks/Java/act/pom.xml

Line 71 in a5aa5b3

<java.version>1.8</java.version>

Take a look here. These are my notes for the commits that updated the project implementations to Java 11. Scroll down and you will see several PR with Java 11 in their names. Act wasn't supporting Java 11 at the time. Also the test implementation of Act is actively maintained by the author of the framework. It seems currently Act is still working on Java 11 support.

Also note that some frameworks were already using Java 12 at the time.

[waghanza]

@zloster yes, you got a point. However, the discussion is NOT about that, but more running on the same versions of a language, here java, which I'm trying to do https://github.com/the-benchmarker/web-frameworks

The global idea is not to say if that is a good or a bad idea, but just to clarify intention, but anyway it can be done ❤️

[zloster]

@waghanza
IMO you are missing that the current state could be viewed as: everything is using Java 11 with the following exceptions:

1. implementations which are too old and major effort will be required to be upgraded (Groovy/Grails);
2. implementations which are not yet compatible (Java/act);
3. implementations which are already using newer runtime environment (these that are on Java 12 and later).

For me it's good enough and the goal running the same version of the language runtime is achieved - most of the implementations are on the same version with the exceptions mentioned above which are few.

Another note that is not related to the current discussion:

which I'm trying to do https://github.com/the-benchmarker/web-frameworks

You should take a closer look to #1220 and #1227. Basically reporting latencies for variable loads (different RPS) is misleading.

[waghanza]

@zloster in deed ...

you point it out, there are some exceptions ... but no document (on end-user side) ❤️

like I told you, the main idea here is to point a lack of information, not to have any judgement or any negativity about work done here

https://github.com/the-benchmarker/web-frameworks has not the same philosophy (lack of documentation for this too ^^), but having different point of views is healthy, no ?

[zloster]

@waghanza
Could you add an edit to your first post and title to clarify this.
Thanks