Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand benchmarks to run on PyPy as well #12174

Open
itamarst opened this issue May 8, 2024 · 2 comments · May be fixed by #12175
Open

Expand benchmarks to run on PyPy as well #12174

itamarst opened this issue May 8, 2024 · 2 comments · May be fixed by #12175

Comments

@itamarst
Copy link
Contributor

itamarst commented May 8, 2024

Currently the codspeed.io run on CPython only, but PyPy is also important.

Running benchmarks on PyPy requires a slightly different runtime approach, insofar as it has a JIT (and future CPython may add JIT too, of course). In particular, my impression is that codspeed will only run the code once, and measure instruction count. This is fine for CPython, but for a JIT it will give misleading results. So the benchmarking framework should run code multiple times if it's on PyPy.

This can be done with a custom pytest fixture.

Separately, there's the issue of how to communicate different runs to codspeed, which I will also look into.

@itamarst
Copy link
Contributor Author

itamarst commented May 8, 2024

@p12tic FYI.

@itamarst itamarst linked a pull request May 8, 2024 that will close this issue
@itamarst itamarst linked a pull request May 8, 2024 that will close this issue
@itamarst
Copy link
Contributor Author

itamarst commented May 8, 2024

OK, as mentioned at CodSpeedHQ/pytest-codspeed#33 PyPy benchmarks are a Problem. Maybe not a problem for Twisted though, so that may not be a blocker, will have to do some more investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant