A lot of repeated json and plaintext tests

[joanhey]

After check it, a lot of frameworks are repeating json and plaintext tests.
Some are for different platforms or libs, and it's OK.
But many only because have them in database variants.

Plaintext repeated:

• Sinatra 14
• Officefloor 11 @sagenschneider
• Falcon 11 @remittor
• Fastapi 11 @micaelmalta
• Chi 7 @0uep
• Flask 7 @remittor
• Roda 6 @timuckun Roda #8160
• Nestjs 6 @JHyeok
• Tornado 4 @Dith3r
• Rails
• Rack
• Microdot
• Cppcms
• Lithium
• ...

Work in progress.

[remittor]

@joanhey , do I need to remove only test falcon [meinheld-orjson] ?
All other tests have different http servers.

[remittor]

@joanhey , PyPy variants (flask-raw & flask) already marked as broken : f8e8239#diff-bf7a33c9cd47aa475c2802732158ab4d7009ed11dafefea8a6114204558925d3R112

[joanhey]

@remittor orjson looks like a different json lib, but the plaintext and fortunes don't use json.

About flask in plaintext and also json:

[joanhey]

I don't know the difference between Sinatra and Sinatra Sequel.

But in the plaintext test, the results are identical.

EDIT:
Sequel is only an ORM, so we can remove all json and plaintext tests.
b2b1b91

[timuckun]

The difference between Sinatra and Sinatra Sequel is the ORM. Sinatra uses ActiveRecord and Sinatra Sequel uses sequel.

Of course neither the JSON nor the plaintext tests touch the database so it doesn't matter in those cases.

One issue with the ruby benchmarks are that they all test mysql and postges and three different servers (puma, unicorn, passenger). Every combination is a different performance profile but I think the puma-postgres combination is likely to be fastest.

Also the ruby tests are quite old (except for rails). I just did a pull request for Roda to update it and will also update the other ones soon. This should make some difference in the performance of the ruby benchmarks.

[joanhey]

@timuckun remember to delete the extra tests in Roda
Thank you

[timuckun]

Which are the extra tests? Do you mean the MySQL tests or the the other servers, or the plaintext/json tests?

It seems to me there is some value in knowing which of the various ruby servers performs best under these loads.

Perhaps a good idea is to test the servers in the currently broken rack tests and then use the fastest one in the rest of the ruby tests. Rack is the underlying library in virtually all the ruby frameworks so that would establish a good baseline for testing various threading/fiber/forking/async tests.

[micaelmalta]

Indeed, the web-server chosen will respond differently against the latency of the application.
So, IMO, it s a good idea to the test all the possible combinations

Maybe, it should be imposed 1 web-server per language (gunicorn for Python for ex)

[joanhey]

If you test a Json lib, you don't need Plaintext and Fortunes, as don't use json.

For every server is OK to have all the tests.
But with the same server and different database, we don't need to repeat the Json and Plaintext.

Examples:

[timuckun]

The way ruby works is that you use the same class for the functionality and use a different invocation based on the server.

For example

bundle exec passenger start ....
bundle exec unicorn ...
bundle exec puma ...

All of these will all read the same config.ru file which will execute whatever class is defined in there. Each docker file is slightly different but the code is the same for all of the servers. The path of least resistance is the repeat all the tests for each server.

I am happy to get rid of the mysql tests but in the cast of ORMs it might be useful to see how the performance of the ORM and the client libs differ between the databases.

As mentioned above here is what I propose.

I would fix the broken ruby/rack tests and test all the servers in that project and then limit the other ruby tests to whichever server is the fastest. Perhaps the rack test could also test both databases using native libs. This would establish a baseline for all ruby based benchmarks as they all depend on rack. Once that baseline is established each framework would only test their own project with one server and one database using whatever ORM or library it comes bundled with. I don't think every server needs to be tested with both databases so I would use the most widely used server (puma) to test both databases the rest would only be tested with postgres.

[joanhey]

Of course that you need to test for every server and database.

But the same server, don't need to test json and plaintext for each database.
We only need to test the database tests.
The json and plaintext will be the same with any database, for that we don't need to repeat.

Example plaintext:

The passenger mri, will have the same results in json and plaintext with any database.
So we only need to test it one time for every server, and not with any database too.

[timuckun]

I understand what you are saying. I can remove it from the benchmarks.json for the additional tests.

[KostyaTretyak]

In the directory frameworks/JavaScript and frameworks/TypeScript there are often unnecessary test duplicates. For example, this applies to nodejs, fastify, etc.

By the way, not only json and plaintext are repeated. This also applies to tests involving databases. In tests with json and plaintext (where benchmark_config -> display_name does not mention any database), it does not make sense to involve the database.

[JHyeok]

@KostyaTretyak
I don't see any repeated tests in nodejs, fastify, and nest except for chakra.

[KostyaTretyak]

@JHyeok, if you see a mix, for example, plaintext_url with db_url - these are duplicates. And there are indeed many, including nodejs, fastify, nest and so on...

It makes no sense to make such a mix, since the results should contain statistics plaintext with json or databases.