ENH add confidence intervals to plotly results #339
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request adds support for confidence intervals to the result plots, but just for the plotly versions of the plots. The idea is to have intervals that show the confidence for the median in addition to quantile plots, so that the width of the intervals decrease as the number of repetitions increase. I've changed the slider to a drop-down list instead. The default is still quantiles.
Here's for instance a benchmark using quantiles and confidence intervals, which was run for 50 repetitions:
I've used standard percentile bootstrap CIs here because they are always positive and need less babysitting that bias-controlled intervals do. The bootstrapping does add a bit of extra overhead, but I've found it to be largely neglible. But if you would prefer an interval that's faster to compute then there are alternatives.
This functionality could be added to the CLI interface too of course, although it would require a new argument.
My proficiency in javascript is horrible by the way, so I hope I didn't screw anything up.