Option to specify desired run time

[dave1010]

Instead of using trial and error to work out useful run times (both for quick feedback during development and longer run times for CI), being able to specify a desired run time would allow PHPBench to collect as many samples as possible in a desired time. PHPBench should be doing this maths for me :-).

For example, when making minor tweaks and in a TDD-style flow I'd want to get results back within 2 or 3 seconds but when using Jenkins I may be prepared to wait up to an hour.

Ideally this would be a global option, and PHPBench would run benchmarks in a round robin style, to get an even spread. PHPBench would have to predict the time based on previous samples, so this would not be 100% accurate but would still be useful.

[dantleech]

I think the issue is that just saying "keep running for 2 minutes" might sometimes result in only 1 or 2 iterations with slow benchmarks - which would not be very indicative. Also when using the "retry theshold" feature the time is unbounded - it takes as long as it takes (although there will probably be a "give up after 10 minutes" feature eventually).

Time specification I think could apply better to the number of times the method is executed within a SINGLE iteration. For example: "keep running this method for 1 minute and tell me how many executions it did within that time". This is what JMH allows you to do.

But then the problem is that you would have to verify on each revolution how much time had elasped -- which would actually change the results of the benchmark.

[dantleech]

Have just had another look at this, JMH achieves this by allowing a time to be specified for every iteration (explanation here). But I am not sure how best to implement this in the current implementation, as checking to see if the time limit has elapsed would affect the measurement (maybe that is acceptable in some if not most situations however).

The other option is to keep running iterations until a given time has elapsed. This is realizable, however, it raises another issue - as you say in Jenkins you want the benchmark to run for 1 hour, but locally not so. So there we have a problem of configuring two different environments.

The time could be overridden on the CLI, but then it would apply to all benchmarks.. but then maybe the user should just launch PHPBench multiple times (and filter the benchmarks) - or we could introduce "suites" in the configuration which allow sets of benchmarks to be defined with options which override the annotated values.

"suites" I think is already a missing feature, so perhaps that is the way to solve this.

[dantleech]

Might be possible with refactorings happening in the Runner in 1.0

[tillkruss]

For us it would be very interesting to benchmark operations per second for each iteration as an alternative metric to “time consumed”.

[dantleech]

You can do this with OutputTimeMode

[tillkruss]

Some thoughts after my chat with @dantleech:

It would be quite nice to have a test run for 5 iterations, each executing for 10 seconds and collecting how many revolutions are possible (ops/sec?)

Alternatively: Maybe the benchmark could run for 5 iteration, executing 10 seconds each, and the benchmark will increate the operations counter itself:

public function benchSomething()
{
    md5('foobar');

    $this->completedIteration(); // +1
}