Add option for a minimum number of samples

[davidanthoff]

It would be great if there was an option (ideally even the default) that allowed me to make sure any benchmarkable code ran at least n times, even if the combined limits of samples and seconds suggest that we only need one sample. I mainly would want this so that one can in a generic way make sure to get a least one run that doesn't include any method compilation time.

[jrevels]

BenchmarkTool's API is budget-based, since that's generally how real-world performance experiments are parameterized (e.g. "I want to take as many samples as possible up to x samples, and I have y amount of time to do so."). If you prescribe enough time to take more than one sample, then BenchmarkTools will definitely take more than one sample.

If you don't care about time, and want a certain number of samples no matter what, you can tell BenchmarkTools that; set the time to an extremely high amount and set the sample budget appropriately.

I mainly would want this so that one can in a generic way make sure to get a least one run that doesn't include any method compilation time.

As long as you tune! the benchmark first (which a lot of the API entry points do for you), this should already be the case.

[rayegun]

I'd like to bump this if possible. It'd be very nice to set an exact number of samples, regardless of seconds, especially when comparing against other languages I find they run a specific number of repeats, which I'd like to match.

[gdalle]

doesn't it work with seconds = Inf?

[101001000]

doesn't it work with seconds = Inf?

I'm trying seconds = Inf currently for a project, but while it performs the amount of samples correctly, it just takes so long than including some arbitrary time limit. For example, if doing samples=10000 seconds=1 takes 10 seconds and performs 2000 samples, one would expect samples=10000 seconds=Inf would take 50 seconds, but is not the case.

Doing samples=10000 seconds=5 take 50 seconds and returns 10000 samples.

Maybe BenchmarkTools by having no time limit is spending more time seeking for quality samples? idk

[gdalle]

Can you share a reproducible example?