[Feature Request] Benchmarks in GH actions

[vmoens]

Motivation

We could create a benchmarking workflow in gh action and log these after very merge.

This would probably involve turning the benchmarks into pytest?

The alert is super cool too!

Ideally we'd like to have one workflow on CPU and one on GPU.

Resources:

• https://github.com/marketplace/actions/continuous-benchmark
• https://pypi.org/project/pytest-benchmark/

@tcbegley pinging you for context. I will create a BC task with this, but feel free to add anything you think is relevant.

[mischab]

I will start working on this, claimed BC task