is @benchmarkset usable ?

[filchristou]

@benchmarkset is nowhere in the documentation and can only be found in the the API Reference.

Is there a reason why ? Like is it obselete or maybe "too new" ?

Anyway, I think it's also malfunctioning.
E.g., I would expect the refdocs example to produce 5 cases but it only produces 1:

julia> @benchmarkset "suite" for k in 1:5
           @case "case $k" rand($k, $k)
       end
1-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "suite" => 1-element BenchmarkTools.BenchmarkGroup:
	 tags: []
	 "case 5" => Benchmark(evals=1, seconds=5.0, samples=10000)

I would expect it to be similar with the following code block:

julia> bg = BenchmarkGroup([], "suite" => BenchmarkGroup([]));

julia> foreach(1:5) do k
           bg["suite"]["case $k"] = @benchmarkable rand($k, $k)
       end

julia> bg
1-element BenchmarkTools.BenchmarkGroup:
  tags: []
  "suite" => 5-element BenchmarkTools.BenchmarkGroup:
	 tags: []
	 "case 2" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 4" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 5" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 3" => Benchmark(evals=1, seconds=5.0, samples=10000)
	 "case 1" => Benchmark(evals=1, seconds=5.0, samples=10000)

System

Julia Version 1.9.4
Commit 8e5136fa297 (2023-11-14 08:46 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 12 × 12th Gen Intel(R) Core(TM) i5-1235U
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-14.0.6 (ORCJIT, alderlake)
Threads: 1 on 12 virtual cores
Environment:
JULIA_IMAGE_THREADS = 1

[32113eaa] PkgBenchmark v0.2.12

similar discussion in https://discourse.julialang.org/t/use-tags-setup-teardown-with-the-benchmarkset-macro/98936/2

[Zentrik]

It seems this was fixed in the latest release, v1.4, in pr #284. It seems like PkgBenchmark restricts you to using an old version of BenchmarkTools, though I'm not 100% sure on that. Either way you need to update BenchmarkTools.