Choosing CPU sampling interval

[sandbubbles]

Hi everyone,
Scalene uses a default sampling interval of 10 milliseconds.
Could someone from the Scalene development team shed some light on why 10 milliseconds was chosen as the default sampling interval?
Are there any recommendations or considerations for adjusting the sampling interval? For example, is there some benchmark that would indicate what is a suitable sampling interval?

[emeryberger]

10ms is pretty standard for sampling profilers: it's the default interval used by Go, VTune, and the async profiler for Java. Py-spy also uses a 10ms rate. We also used 10ms in our Coz causal profiler.

This rate empirically strikes a sweet spot providing high accuracy and low overhead (for Scalene, see the graph of Figure 5 from our OSDI paper demonstrating that Scalene's choice delivers high accuracy).

There are a few notable departures from this default: v8's profiler uses a 1ms interval. I just tried perf and while the linked page says it uses a 1ms interval, on my system, it was more like 0.2ms. Austin also uses a default of 0.1ms.

[sandbubbles]

Thank you for the explanation!
Am I correct in understanding that this decision doesn't necessarily need to consider factors such as the speed at which GIL interprets instructions or Scalene's maximum processing pace for signals and samples?

Furthermore, Scalene adds to both counters at each sample - for Python execution, it is increased by the sampling interval and for native execution, it is increased by the difference between the elapsed time and the sampling interval. Is this approach designed to compensate for the lack of precise knowledge about the exact moment of transition from Python execution to native code?

[emeryberger]

First, a clarification: the GIL is just a lock (the _G_lobal _I_nterpreter _L_ock); it's the Python interpreter that interprets bytecodes.

Second, Scalene actually only samples at a rate of at most once per 10ms. It restarts the timer after it processes each sample. This implicitly adjusts for processing time.

And yes, its use of two separate counters lets it tease apart time spent in the Python interpreter vs. time spent in native code.

[emeryberger]

I should add that the latter part is covered in detail in this technical paper (OSDI 2023).