Relative difference for throughput

[tillkruss]

When using the throughput output the relative difference isn't working.

{
  "expressions": {
    "diff": "format(\"%.1fx (%.1f%%)\", mode(suite[subject_groups = [\"predis\"]][\"result_time_avg\"]) / mode(result_time_avg), percent_diff(mode(result_time_avg), mode(suite[subject_groups = [\"predis\"]][\"result_time_avg\"])))"
  }
}

When I run: ./vendor/bin/phpbench run --report=redis --filter=GetThroughputBench it works:

./vendor/bin/phpbench run --report=redis --filter=GetThroughputBench
PHPBench (1.2.6) running benchmarks... #standwithukraine
with configuration file: /Users/Till/Development/Repositories/relay/phpbench.json
with PHP version 8.1.11, xdebug ❌, opcache ❌

\CacheWerk\Relay\Benchmarks\Throughput\GetThroughputBench

    GET_Throughput_Predis...................I1 - Mo54,546ops/s (±2.24%)
    GET_Throughput_Credis...................I1 - Mo31,919ops/s (±1.16%)
    GET_Throughput_PhpRedis.................I1 - Mo33,001ops/s (±1.08%)
    GET_Throughput_Relay_NoCache............I1 - Mo59,426ops/s (±0.99%)
    GET_Throughput_Relay_WarmCache..........I1 - Mo4,934,810ops/s (±1.15%)

Subjects: 5, Assertions: 0, Failures: 0, Errors: 0
GetThroughputBench
+--------------------------------+---------+-----+--------+-----------+----------------+-----------------+
| subject                        | revs    | its | rstdev | mem_peak  | mode           | diff            |
+--------------------------------+---------+-----+--------+-----------+----------------+-----------------+
| GET_Throughput_Predis          | 50000   | 2   | ±2.24% | 1.073mb   | 54,546ops/s    | 1.0x (0.0%)     |
| GET_Throughput_Credis          | 50000   | 2   | ±1.16% | 1.208mb   | 31,919ops/s    | 0.6x (-41.5%)   |
| GET_Throughput_PhpRedis        | 50000   | 2   | ±1.08% | 808.544kb | 33,001ops/s    | 0.6x (-39.5%)   |
| GET_Throughput_Relay_NoCache   | 50000   | 2   | ±0.99% | 808.560kb | 59,426ops/s    | 1.1x (8.9%)     |
| GET_Throughput_Relay_WarmCache | 5000000 | 2   | ±1.15% | 808.560kb | 4,934,810ops/s | 90.5x (8947.1%) |
+--------------------------------+---------+-----+--------+-----------+----------------+-----------------+

But just running ./vendor/bin/phpbench run --report=redis skews it up:

./vendor/bin/phpbench run --report=redis
PHPBench (1.2.6) running benchmarks... #standwithukraine
with configuration file: /Users/Till/Development/Repositories/relay/phpbench.json
with PHP version 8.1.11, xdebug ❌, opcache ❌

\CacheWerk\Relay\Benchmarks\MgetBench

    MGET_Predis.............................I1 - Mo4.953ms (±1.01%)
    MGET_Credis.............................I1 - Mo7.148ms (±0.11%)
    MGET_PhpRedis...........................I1 - Mo6.127ms (±1.48%)
    MGET_Relay_NoCache......................I1 - Mo3.678ms (±0.54%)
    MGET_Relay_WarmCache....................I1 - Mo0.290ms (±3.29%)

\CacheWerk\Relay\Benchmarks\Get\GetStringsBench

    GET_Strings_Predis......................I1 - Mo38.305ms (±0.60%)
    GET_Strings_Credis......................I1 - Mo66.388ms (±1.17%)
    GET_Strings_PhpRedis....................I1 - Mo64.149ms (±0.56%)
    GET_Strings_Relay_NoCache...............I1 - Mo34.167ms (±0.36%)
    GET_Strings_Relay_WarmCache.............I1 - Mo0.301ms (±0.58%)

\CacheWerk\Relay\Benchmarks\Get\GetUnserializeBench

    GET_Unserialize_Predis..................I1 - Mo39.697ms (±1.13%)
    GET_Unserialize_Credis..................I1 - Mo69.264ms (±1.65%)
    GET_Unserialize_PhpRedis................I1 - Mo70.536ms (±0.89%)
    GET_Unserialize_Relay_NoCache...........I1 - Mo36.880ms (±1.91%)
    GET_Unserialize_Relay_WarmCache.........I1 - Mo0.484ms (±2.52%)

\CacheWerk\Relay\Benchmarks\Throughput\GetThroughputBench

    GET_Throughput_Predis...................I1 - Mo54,231ops/s (±3.38%)
    GET_Throughput_Credis...................I1 - Mo30,230ops/s (±1.04%)
    GET_Throughput_PhpRedis.................I1 - Mo33,483ops/s (±0.64%)
    GET_Throughput_Relay_NoCache............I1 - Mo59,335ops/s (±0.47%)
    GET_Throughput_Relay_WarmCache..........I1 - Mo4,958,762ops/s (±1.01%)

Subjects: 20, Assertions: 0, Failures: 0, Errors: 0
MgetBench
+----------------------+------+-----+--------+----------+---------+-------------------+
| subject              | revs | its | rstdev | mem_peak | mode    | diff              |
+----------------------+------+-----+--------+----------+---------+-------------------+
| MGET_Predis          | 50   | 2   | ±1.01% | 1.342mb  | 4.953ms | 7.7x (669.3%)     |
| MGET_Credis          | 50   | 2   | ±0.11% | 1.476mb  | 7.148ms | 5.3x (433.1%)     |
| MGET_PhpRedis        | 50   | 2   | ±1.48% | 1.074mb  | 6.127ms | 6.2x (522.0%)     |
| MGET_Relay_NoCache   | 50   | 2   | ±0.54% | 1.074mb  | 3.678ms | 10.4x (936.1%)    |
| MGET_Relay_WarmCache | 50   | 2   | ±3.29% | 1.074mb  | 0.290ms | 131.3x (13026.5%) |
+----------------------+------+-----+--------+----------+---------+-------------------+

GetStringsBench
+-----------------------------+------+-----+--------+----------+----------+-------------------+
| subject                     | revs | its | rstdev | mem_peak | mode     | diff              |
+-----------------------------+------+-----+--------+----------+----------+-------------------+
| GET_Strings_Predis          | 10   | 2   | ±0.60% | 1.330mb  | 38.305ms | 1.0x (-0.5%)      |
| GET_Strings_Credis          | 10   | 2   | ±1.17% | 1.465mb  | 66.388ms | 0.6x (-42.6%)     |
| GET_Strings_PhpRedis        | 10   | 2   | ±0.56% | 1.065mb  | 64.149ms | 0.6x (-40.6%)     |
| GET_Strings_Relay_NoCache   | 10   | 2   | ±0.36% | 1.065mb  | 34.167ms | 1.1x (11.5%)      |
| GET_Strings_Relay_WarmCache | 10   | 2   | ±0.58% | 1.065mb  | 0.301ms  | 126.5x (12549.1%) |
+-----------------------------+------+-----+--------+----------+----------+-------------------+

GetUnserializeBench
+---------------------------------+------+-----+--------+----------+----------+-----------------+
| subject                         | revs | its | rstdev | mem_peak | mode     | diff            |
+---------------------------------+------+-----+--------+----------+----------+-----------------+
| GET_Unserialize_Predis          | 10   | 2   | ±1.13% | 1.331mb  | 39.697ms | 1.0x (-4.0%)    |
| GET_Unserialize_Credis          | 10   | 2   | ±1.65% | 1.467mb  | 69.264ms | 0.6x (-45.0%)   |
| GET_Unserialize_PhpRedis        | 10   | 2   | ±0.89% | 1.065mb  | 70.536ms | 0.5x (-46.0%)   |
| GET_Unserialize_Relay_NoCache   | 10   | 2   | ±1.91% | 1.065mb  | 36.880ms | 1.0x (3.3%)     |
| GET_Unserialize_Relay_WarmCache | 10   | 2   | ±2.52% | 1.065mb  | 0.484ms  | 78.8x (7775.8%) |
+---------------------------------+------+-----+--------+----------+----------+-----------------+

GetThroughputBench
+--------------------------------+---------+-----+--------+-----------+----------------+-------------------------+
| subject                        | revs    | its | rstdev | mem_peak  | mode           | diff                    |
+--------------------------------+---------+-----+--------+-----------+----------------+-------------------------+
| GET_Throughput_Predis          | 50000   | 2   | ±3.38% | 1.073mb   | 54,231ops/s    | 2066.5x (206546.3%)     |
| GET_Throughput_Credis          | 50000   | 2   | ±1.04% | 1.208mb   | 30,230ops/s    | 1151.9x (115091.6%)     |
| GET_Throughput_PhpRedis        | 50000   | 2   | ±0.64% | 808.544kb | 33,483ops/s    | 1275.9x (127485.1%)     |
| GET_Throughput_Relay_NoCache   | 50000   | 2   | ±0.47% | 808.560kb | 59,335ops/s    | 2261.0x (225995.5%)     |
| GET_Throughput_Relay_WarmCache | 5000000 | 2   | ±1.01% | 808.560kb | 4,958,762ops/s | 188953.7x (18895269.2%) |
+--------------------------------+---------+-----+--------+-----------+----------------+-------------------------+

Maybe related to #974. The code: https://github.com/cachewerk/relay

[dantleech]

what's the exact problem? what part of the expression is returning the wrong results?

[tillkruss]

The issue is that the diff column only works when I only run the GetThroughputBench class, not when running multiple bench cases at once.

Is it possible to define an order of the benchmark classes to run in?

[dantleech]

I assume because it's running the "diff" query on the entire dataset instead of the partition... not really sure who's issue that is without diving into it.

You can use the https://phpbench.readthedocs.io/en/latest/report-components/section.html to partition the data and create a table_aggregate report on each partition, that would guarantee the correct result I think

[tillkruss]

I don't even know where to start 😬🙈

Can you point me at an example?

{
    "report.generators": {
        "redis": {
            "extends": "aggregate",
            "generator": "expression",
            "break": ["benchmark"],
            "expressions": {
                "diff": "format(\"%.1fx (%.1f%%)\", mode(suite[subject_groups = [\"predis\"]][\"result_time_avg\"]) / mode(result_time_avg), percent_diff(mode(result_time_avg), mode(suite[subject_groups = [\"predis\"]][\"result_time_avg\"])))"
            },
            "cols": [
                "benchmark",
                "subject",
                "revs",
                "its",
                "rstdev",
                "mem_peak",
                "mode",
                "diff"
            ]
        }
    }
}