Standardising Benchmarks, with support for nanobench as an option for its backend

[vrnimje]

Background context: #6357

Proposed Changes

• Added bench-marking interface perftests_report, which also supports nanobench as the backend
• Added option for using nanobench, or default to timed-loop, as the back-end for performance measurement
• Fixed compilation issues with the above mentioned PR

[StellarBot]

Can one of the admins verify this patch?

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.00%	✅ 74.00%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (eb2bf57)	217975	185524	85.11%
Head commit (2575d63)	190903 (-27072)	162490 (-23034)	85.12% (+0.00%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#6448)	100	74	74.00%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

_{Codacy will stop sending the deprecated coverage status from June 5th, 2024. Learn more}

[isidorostsa]

Congratulations for getting nanobench into the build system!
I left some minor corrections and a suggestion, do not worry about the tests, they are not failing because of your changes.
Great PR overall :)

[vrnimje]

Congratulations for getting nanobench into the build system! I left some minor corrections and a suggestion, do not worry about the tests, they are not failing because of your changes. Great PR overall :)

Thanks for the response, but I am unable to see your suggestions in the PR. Could you please check once from your side?

[isidorostsa]

Here you could use Nanobench_ROOT to prevent cmake from complaining

[isidorostsa]

Did this not work initially?

[vrnimje]

No. I was just testing by just using this call, could nanobench be inserted into the system.

[isidorostsa]

Do you think you should uncomment "NAMESPACE" and "add_library" to follow the namespace+alias as in other files?

e.g.

hpx/cmake/HPX_SetupAsio.cmake

Line 76 in 72ca840

add_library(Asio::asio ALIAS asio)

[isidorostsa]

This format is suited for internal consumption e.g. to be used along with a python plotting tool. Could we use a template to output the average in a human readable format?

[vrnimje]

Yes. I think then we could make use of an command line option in the benchmarks to invoke the internal template.

[isidorostsa]

It may be possible to do this with a global variable through the hpx command line arguments facilities, e.g. without having to modify benchmarks to pass the template as an explicit variable, I will get back to you when I find out.

[vrnimje]

Ohh, so something like --hpx:threads
Will look into this

hpx/libs/full/command_line_handling/src/command_line_handling.cpp

Lines 248 to 289 in 72ca840

	if (vm.count("hpx:threads"))
	{
	threads_str = vm["hpx:threads"].as<std::string>();
	if ("all" == threads_str)
	{
	threads = init_threads;
	if (batch_threads != static_cast<std::size_t>(-1))
	{
	threads = batch_threads;
	}
	}
	else if ("cores" == threads_str)
	{
	threads = init_cores;
	if (batch_threads != static_cast<std::size_t>(-1))
	{
	threads = batch_threads;
	}
	}
	else
	{
	threads = hpx::util::from_string<std::size_t>(threads_str);
	}
	if (threads == 0)
	{
	throw hpx::detail::command_line_error(
	"Number of --hpx:threads must be greater than 0");
	}
	#if defined(HPX_HAVE_MAX_CPU_COUNT)
	if (threads > HPX_HAVE_MAX_CPU_COUNT)
	{
	// clang-format off
	throw hpx::detail::command_line_error("Requested more than "
	HPX_PP_STRINGIZE(HPX_HAVE_MAX_CPU_COUNT)" --hpx:threads "
	"to use for this application, use the option "
	"-DHPX_WITH_MAX_CPU_COUNT=<N> when configuring HPX.");
	// clang-format on
	}
	#endif
	}

[vrnimje]

It may be possible to do this with a global variable through the hpx command line arguments facilities, e.g. without having to modify benchmarks to pass the template as an explicit variable, I will get back to you when I find out.

I have added a hpx:verbose_bench command line option in dd126c6 for the same. Here is the corresponding CI run testing it: test-run