Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FR] Consider adding extra info to the reducing variance doc relating to CPU frequency scaling #1684

Open
simonhf opened this issue Oct 23, 2023 · 0 comments

Comments

@simonhf
Copy link

simonhf commented Oct 23, 2023

Is your feature request related to a problem? Please describe.

I'm trying to reduce variance in a benchmark, and doing all the things listed in the "reducing variance" doc, here [1].
This works fine on the box on my desk (where I have full control), but when I try and run it in the cloud then variance is back.
Why? AFAICT it's because the one "reducing variance" measure which does not appear to work with most cloud providers is pinning CPU frequency scaling.
And it doesn't seem to make a difference whether it's a regular cloud VM, or a cloud "bare metal" box.
I'm guessing this is because if you allow users to manipulate the CPU frequencies any way they like, then there is a risk of overheating the box?

Describe the solution you'd like

Add a section to the "reducing variance" doc [1], discussing the difficulties and possible solutions of implementing the measures locally versus in the cloud.

Describe alternatives you've considered

Have been Googling around a lot trying to find confirming or deny that "CPU frequency scaling" in some cloud provider somewhere works or doesn't work. Or that maybe local (non-cloud) benchmorks are the only way to go if CPU frequency scaling is needed?

Additional context

n/a

[1] https://github.com/google/benchmark/blob/main/docs/reducing_variance.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant