Add asv benchmarks to track performance

[benjeffery]

Unit tests give us confidence in correctness, this would give us confidence in performance. See for example numpy's: https://pv.github.io/numpy-bench/#/

It would be work to develop the benchmarks, but we could start small.

[petrelharp]

This would be great. @brnorris03 may have input here.

[jeromekelleher]

Yes, this is a great idea! We could start with a handful of things like timing tree iteration and build from there.

My only question is, where do we run them and where do we keep the benchmarks? Can we do this as part of CI?

[benjeffery]

Yep, this would be done as an automated process. I haven't looked into if it's easy to get metrics on branches so you can check before merge, but don't think that's a deal breaker. Key obviously is to have a consistent environment, might need a cheap server somewhere unless I can find a "free for OSS" solution.

[jeromekelleher]

Happy to spend some money on an AWS (or whatever) instance we spin up for this (just need to figure out how we set up the billing with Oxford).

[benjeffery]

I think it would be pretty cheap as it would only spin-up when there are tests to be run.

[brnorris03]

Cloud resources are typically shared (dedicated is possible, but more expensive), so timing would be unreliable. Would it be acceptable to set up local server performance testing instead?

[brnorris03]

More specifically, it is actually not too hard to set up a Travis CI server locally (so I am told, I haven't actually done it), which would enable us to use Travis for both the correctness and performance tests. Ideally, you'd have a few different architectures running CI servers.

[jeromekelleher]

Thanks @brnorris03, that's really helpful. We could set up a local server and just run these benchmarks as a pre-release exercise. If we want to run this for every PR (and, I guess, you'd really want to do this to catch regressions early) we'd have to make it a cloud setup.

How do numpy set theirs up @benjeffery?

[benjeffery]

I can't find anything on where numpy run theirs, the output is uploaded to a github pages site at https://github.com/pv/numpy-bench

Pandas run theirs in someone's basement! https://tomaugspurger.github.io/maintaing-performance.html

I'll look into options when I tackle this ticket.

[benjeffery]

We now have these in msprime, so can use the same infrastructure.