Automate installation of the iai-callgrind-runner binary in case it cannot be found in the PATH

[Joining7943]

Currently, when running a iai-callgrind benchmark, the path to the iai-callgrind-runner binary is determined by

1. Trying the IAI_CALLGRIND_RUNNER environment variable
2. Trying PATH

and iai-callgrind benchmarks fail if the binary cannot be found in the PATH. Both options rely on user interaction to install the binary. Managing the installation of the binary can be automated and especially setting up iai-callgrind in the ci would be simplified. Also, updating the library requires user interaction to also update the runner to the exact same version. The following procedure could solve these issues (IAI_CALLGRIND_VERSION is the library version):

1. Try the IAI_CALLGRIND_RUNNER environment variable
2. Try the PATH
3. Try $XDG_CACHE_DIR/iai-callgrind/$IAI_CALLGRIND_VERSION/bin
4. Try $HOME/.cache/iai-callgrind/$IAI_CALLGRIND_VERSION/bin
5. Try target/iai-callgrind/$IAI_CALLGRIND_VERSION/bin

If all of these directories don't contain the iai-callgrind-runner binary, automatically install the binary in the path 3. - 5. depending on the existence of XDG_CACHE_DIR or $HOME/.cache. 5. is the safe fallback.

[Joining7943]

To opt-out of the automatic installation an environment variable IAI_CALLGRIND_RUNNER_INSTALL or something like that would be great. The variable has the default value auto but also accepts values off, no etc. to switch off automatic installation of the binary.

[mohe2015]

Have you thought about some magic like letting the main! macro expand to a check for a command line flag or so that then branches to the code for the callgrind runner? So the main! macro first runs its normal code and then forks itself with a new command line parameter and then runs the callgrind runner?
Or does that not work for some reason or is too inaccurate for benchmarking?

I would really love a solution where you don't need to do any manual steps but I also don't really like automatic install. It would probably depend on things from the host that the user then can't control that easily any more.

[Joining7943]

Thanks for sharing your opinion :)

A problem with passing command line arguments to the benchmarks via cargo bench ... -- ARGS is, that they only work when running a benchmark with the --bench flag. An environment variable on the other hand would always work. Also, adding code to the macros (main!, library_benchmark_groups! ...) may have an impact on library benchmarks and indeed would make them more inaccurate. That's the reason why most of the code has been moved into a separate binary; to be separated from the to be benchmarked binary with the library benchmarks in it. Btw, binary benchmarks are not affected, since the benchmarked binary is passed directly to valgrind.

Currently, I tend to making the automatic installation opt-in instead of opt-out. Maybe this'll solve your concerns? There are mainly two use cases I would like to address. First, simplify running and installing iai-callgrind in the CI, docker etc. and 2nd running iai-callgrind when there are multiple local repos with different versions of iai-callgrind.

It would probably depend on things from the host that the user then can't control that easily any more.

The automatic installation routine would make use of cargo and, per default, install the binary in the temporary locations $HOME/.cache and the target directory as a fallback. This installation option would be documented in the installation section of the README right at the start. So, I hope it's a reasonable solution for someone who wants to use it, especially when it'll be an opt-in solution via an environment variable.