Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

List of 50 longest TFB tests #8131

Open
remittor opened this issue Apr 6, 2023 · 6 comments
Open

List of 50 longest TFB tests #8131

remittor opened this issue Apr 6, 2023 · 6 comments

Comments

@remittor
Copy link
Contributor

remittor commented Apr 6, 2023 •
edited

TFB results of 2023-04-06: https://www.techempower.com/benchmarks/#section=test&runid=ab65d6b5-8efc-4a65-9c5c-c31667418528

Full stat of tests: https://tfb-status.techempower.com/raw/results.2023-04-06-08-48-50-340.zip (56MB)

Parsed stats:
Format: total_test_time (build_time + verify_time) test_name

$ python3 tfb_res.py -f results.2023-04-06-08-48-50-340.zip
Number of frameworks: 295
Number of tests: 767
Total time: 6 days, 15:26:25 (included 12.78 hours of sleep)
Total time: 146.66 + 12.78 = 159.44 hours
Total build  time: 27.65 hours
Total verify time: 114.96 hours
Verify time avg: (114.96 * 60) / 767 = 8.99 min
Number of full tests: 312
Full tests verify time avg: (68.03 * 60) / 312 = 13.08 min

0:39:10 (0:32:35 + 0:06:17)  wizzardo-inline
0:33:28 (0:02:48 + 0:30:27)  viz-diesel
0:29:23 (0:16:10 + 0:12:59)  warp
0:29:01 (0:15:53 + 0:12:58)  drogon
0:28:41 (0:15:19 + 0:12:59)  warp-mysql-haskell
0:28:30 (0:15:17 + 0:12:59)  warp-hasql
0:27:06 (0:14:42 + 0:11:12)  spliffy-mongodb
0:26:34 (0:22:28 + 0:03:59)  servant
0:26:19 (0:21:32 + 0:04:33)  servant-psql-simple
0:26:11 (0:14:44 + 0:11:12)  spliffy-postgres
0:26:10 (0:14:37 + 0:11:11)  spliffy-mysql
0:25:18 (0:00:36 + 0:24:15)  hapi-nginx
0:25:12 (0:16:02 + 0:08:57)  aspcore-aot-ado-pg
0:25:06 (0:08:42 + 0:15:15)  treefrog-mongodb
0:25:03 (0:15:52 + 0:08:58)  drogon-core
0:24:52 (0:09:23 + 0:15:16)  treefrog
0:24:30 (0:14:53 + 0:09:19)  spring-webflux-rxjdbc
0:24:21 (0:08:43 + 0:15:16)  treefrog-mysql
0:24:16 (0:08:48 + 0:15:15)  treefrog-epoll
0:24:15 (0:01:18 + 0:22:42)  rack-sequel-postgres-passenger-mri
0:22:39 (0:01:17 + 0:20:56)  rack-sequel-passenger-mri
0:22:35 (0:08:49 + 0:13:34)  userver
0:22:22 (0:13:11 + 0:08:58)  ntex-db-astd
0:22:07 (0:08:44 + 0:13:01)  phalcon
0:22:00 (0:06:40 + 0:15:10)  anansi
0:21:49 (0:10:27 + 0:11:02)  cppcms
0:21:40 (0:10:40 + 0:10:45)  cppcms-postgres
0:21:31 (0:10:10 + 0:11:10)  appmpower-ado-pg
0:21:24 (0:06:02 + 0:15:09)  userver-bare
0:19:52 (0:06:31 + 0:13:00)  lumen-workerman
0:19:50 (0:08:41 + 0:10:46)  phalcon-micro
0:19:46 (0:06:23 + 0:13:01)  lumen-laravel-s
0:19:34 (0:06:15 + 0:12:59)  mixphp-swoole-mysql
0:19:30 (0:06:18 + 0:12:58)  ubiquity-swoole
0:19:27 (0:03:55 + 0:15:09)  elixir-plug-ecto
0:19:27 (0:08:14 + 0:11:03)  ihp
0:19:21 (0:05:57 + 0:13:00)  laravel-swoole
0:19:21 (0:06:08 + 0:13:00)  lucky
0:19:20 (0:05:56 + 0:13:01)  laravel-laravel-s
0:19:02 (0:03:41 + 0:15:09)  h2o
0:18:57 (0:14:51 + 0:03:59)  spliffy
0:18:56 (0:05:39 + 0:13:01)  ningle
0:18:55 (0:05:35 + 0:12:58)  one-no-coroutine
0:18:40 (0:14:36 + 0:03:59)  0http
0:18:39 (0:05:18 + 0:12:58)  swoole
0:18:38 (0:05:15 + 0:13:00)  lumen
0:18:38 (0:05:20 + 0:13:01)  woo
0:18:36 (0:03:03 + 0:15:10)  cfml
0:18:28 (0:05:06 + 0:13:00)  lumen-swoole
0:18:28 (0:05:18 + 0:12:58)  rocket

Parser source code: https://gist.github.com/remittor/38ef89e47c18f6ec2a2095bc7390cdf3
@remittor
Copy link
Contributor Author

remittor commented Apr 6, 2023 •
edited

Parsed stats for frameworks:
Format: total_test_time (build_time + verify_time) framework_name

$ python3 tfb_res.py --fw -f results.2023-04-06-08-48-50-340.zip
Number of frameworks: 295
Number of tests: 767
Total time: 6 days, 15:26:25 (included 12.78 hours of sleep)
Total time: 146.66 + 12.78 = 159.44 hours
Total build  time: 27.65 hours
Total verify time: 114.96 hours
Verify time avg: (114.96 * 60) / 767 = 8.99 min
Number of full tests: 312
Full tests verify time avg: (68.03 * 60) / 312 = 13.08 min

3:32:39 (0:16:58 + 3:12:51)  http4k
2:52:36 (0:09:14 + 2:40:22)  officefloor
2:25:48 (0:39:08 + 1:42:56)  ubiquity
2:17:17 (0:54:02 + 1:17:45)  cutelyst
2:16:51 (0:23:52 + 1:49:07)  php
2:12:59 (0:08:39 + 2:01:38)  rack-sequel
2:04:03 (0:17:55 + 1:43:27)  falcon
2:02:45 (0:14:35 + 1:46:33)  hexagon
1:57:36 (0:28:35 + 1:24:59)  micronaut   ERROR: "Problem starting micronaut-graalvm"
1:51:30 (0:11:01 + 1:35:05)  play2
1:51:19 (0:08:19 + 1:40:15)  sinatra-sequel
1:48:03 (0:08:58 + 1:33:13)  pippo
1:47:13 (0:07:50 + 1:36:32)  fastapi
1:43:48 (0:17:06 + 1:24:16)  imi
1:42:50 (0:07:54 + 1:31:18)  nestjs
1:41:11 (0:12:22 + 1:26:59)  ktor   ERROR: "Problem starting ktor-exposed-dao"
1:40:52 (0:31:50 + 1:05:01)  spring
1:38:35 (0:35:36 + 1:01:02)  treefrog
1:38:24 (0:58:54 + 0:37:34)  spliffy
1:31:33 (0:45:54 + 0:42:07)  axum
1:29:46 (0:02:07 + 1:25:01)  chi
1:28:21 (0:21:26 + 1:05:01)  laravel
1:26:45 (0:06:51 + 1:17:58)  roda-sequel
1:26:34 (0:46:46 + 0:38:57)  warp
1:20:54 (0:34:48 + 0:43:50)  actix
1:19:36 (0:07:30 + 1:10:09)  sinatra
1:16:44 (0:23:15 + 0:52:01)  lumen
1:14:00 (0:02:25 + 1:09:07)  ffead-cpp
1:09:10 (0:01:16 + 1:04:12)  nodejs
1:06:18 (0:28:01 + 0:37:30)  appmpower
1:05:01 (0:03:00 + 1:00:56)  jooby
1:04:46 (0:11:37 + 0:52:22)  viz
1:03:35 (0:14:23 + 0:47:55)  mixphp
0:59:41 (0:28:26 + 0:29:53)  ntex   ERROR: "Problem starting ntex-plt-astd"
0:59:11 (0:01:35 + 0:55:24)  hapi
0:56:06 (0:07:04 + 0:45:33)  vertx-web   ERROR: "Framework is not accepting requests from client machine"
0:54:50 (0:09:39 + 0:43:55)  kumbiaphp
0:54:04 (0:31:45 + 0:21:56)  drogon
0:54:00 (0:05:10 + 0:45:30)  lithium
0:52:53 (0:44:00 + 0:08:32)  servant
0:52:28 (0:02:45 + 0:48:44)  microdot
0:51:16 (0:13:51 + 0:36:06)  roa
0:49:57 (0:03:48 + 0:45:28)  redkale
0:47:44 (0:15:51 + 0:30:55)  swoole
0:47:43 (0:07:07 + 0:39:48)  xitca-web
0:46:54 (0:22:56 + 0:21:59)  vapor
0:46:31 (0:14:36 + 0:30:55)  php-ngx
0:46:13 (0:14:20 + 0:30:57)  openswoole
0:44:53 (0:05:02 + 0:38:57)  quarkus
0:44:28 (0:08:34 + 0:34:55)  slim

@chrislearn
Copy link
Contributor

Rust framework always much time on build docker images . Is taht possible to use docker muilt-build stage or shared a comon build image or other way to share the compile results?

@remittor
Copy link
Contributor Author

remittor commented Apr 7, 2023 •
edited

Let's take a look at this:
0:33:28 (0:02:48 + 0:30:27) viz-diesel
Verify time = 30.5 min (avg time = 13 min)

Let's look at the details for test fortunes:
https://tfb-status.techempower.com/unzip/results.2023-04-06-08-48-50-340.zip/results/20230330235259/viz-diesel/fortune

 Concurrency: 16 for fortune
 wrk -H 'Host: 10.0.0.1' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 16 --timeout 8 -t 16 "http://10.0.0.1:8080/fortunes"
---------------------------------------------------------
unable to connect to 10.0.0.1:8080 Connection timed out
STARTTIME 1680761646
ENDTIME 1680761776

All fortunes tests failed and took a lot of time: 776 - 646 = 130 sec (2.17 min)
2.17 min * 8 tests = 17.33 min

Checking the result by the time the file was created stat.txt and raw.txt:
1680751642 - 1680750576 = 1066 sec = 17.76 min

Ideally, each test should take 15 seconds.
So the ideal total time for fortune test should be: 15 * 8 = 120 seconds (2 min).

But viz-diesel fortunes test somehow managed to pass the test:

--------------------------------------------------------------------------------
VERIFYING FORTUNE
--------------------------------------------------------------------------------
   PASS for http://10.0.0.1:8080/fortunes
     Executed queries: 512/512
   PASS for http://10.0.0.1:8080/fortunes
     Rows read: 6077/6144

Test pre-verification time: 30.5 min - 17.76 min = 12.74 min

Verify test in CI: https://github.com/TechEmpower/FrameworkBenchmarks/actions/runs/4554118663/jobs/8031564280#step:9:876

UPDATE
Full stat for this test:

$ python3 tfb_res.py -f results.2023-04-06-08-48-50-340.zip -t viz-diesel
Number of frameworks: 295
Number of tests: 767
Total time: 6 days, 15:26:25 (included 12.78 hours of sleep)
Total time: 146.66 + 12.78 = 159.44 hours
Total build  time: 27.65 hours
Total verify time: 114.96 hours
Verify time avg: (114.96 * 60) / 767 = 8.99 min
Number of full tests: 316
Full tests verify time avg: (68.86 * 60) / 316 = 13.07 min

0:33:28 (0:02:48 + 0:30:27)  viz-diesel

                  verify    test
             db : 0:00:04   0:02:12
          query : 0:00:16   0:01:54
         update : 0:00:22   0:07:42
        fortune : 0:00:04   0:17:46

          TOTAL : 0:00:46   0:29:34

Total verify and test time: 0:30:20

@remittor
Copy link
Contributor Author

remittor commented Apr 7, 2023 •
edited

Let's take a look at this:
0:24:15 (0:01:18 + 0:22:42) rack-sequel-postgres-passenger-mri
Verify time = 22.7 min (avg time = 13 min)

Let's look at the details for test fortunes:
https://tfb-status.techempower.com/unzip/results.2023-04-06-08-48-50-340.zip/results/20230330235259/rack-sequel-postgres-passenger-mri

Full stat for this test:

$ python3 tfb_res.py -f results.2023-04-06-08-48-50-340.zip -t rack-sequel-postgres-passenger-mri
Number of frameworks: 295
Number of tests: 767
Total time: 6 days, 15:26:25 (included 12.78 hours of sleep)
Total time: 146.66 + 12.78 = 159.44 hours
Total build  time: 27.65 hours
Total verify time: 114.96 hours
Verify time avg: (114.96 * 60) / 767 = 8.99 min
Number of full tests: 316
Full tests verify time avg: (68.86 * 60) / 316 = 13.07 min

0:24:15 (0:01:18 + 0:22:42)  rack-sequel-postgres-passenger-mri

                  verify    test
             db : 0:00:06   0:04:04
           json : 0:00:04   0:05:22
          query : 0:00:16   0:02:58
         update : 0:00:16   0:03:00
        fortune : 0:00:04   0:03:54
      plaintext : 0:00:04   0:02:24

          TOTAL : 0:00:50   0:21:42

Total verify and test time: 0:22:32

Look to /json/raw.txt : https://tfb-status.techempower.com/unzip/results.2023-04-06-08-48-50-340.zip/results/20230330235259/rack-sequel-postgres-passenger-mri/json/raw.txt

 Concurrency: 256 for json
 wrk -H 'Host: 10.0.0.1' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 15 -c 256 --timeout 8 -t 28 "http://10.0.0.1:8080/json"
---------------------------------------------------------
Running 15s test @ http://10.0.0.1:8080/json
  28 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.12ms    1.02ms  11.65ms   79.87%
    Req/Sec     1.12k   159.09     1.26k    89.29%
  Latency Distribution
     50%    6.13ms
     75%    6.67ms
     90%    7.26ms
     99%    8.76ms
  3110 requests in 15.09s, 735.41KB read
Requests/sec:    206.16
Transfer/sec:     48.75KB
STARTTIME 1680629011
ENDTIME 1680629091

1680629091 - 1680629011 = 80 sec

@svetlyak40wt
Copy link
Contributor

@remittor where did you get tfb_res.py? I can't find it in the repository.

I'm looking the way to output total stats for results of tests started in my local environment. Ideally if it will be possible to view a comparison between two runs. Is there such tool?

@remittor
Copy link
Contributor Author

@svetlyak40wt , look first post
https://gist.github.com/remittor/38ef89e47c18f6ec2a2095bc7390cdf3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@svetlyak40wt @chrislearn @remittor