[Feature request] A second cross depending on the evaluations sum from all trials

[nikohansen]

When a solver terminates without relying on the experimentation budget, the number of independent trials is relevant for the number successes we expect to see. A single success is enough to solve the function+dimension via (simulated) restarts. For this reason, it would be useful know the sum of all evaluations from all trials for a given function+dimension.

Open questions: which marker should we use? What should we do if the sum is larger than the x-limit? Does it make sense to divide by the number of different instances too (a third symbol?)? Or does it make sense to divide by the number of successes too, that is, have an ERT marker?

[nikohansen]

Let runtime refer to successful and unsuccessful trials here, then we have

• median runtime (see also median budget cross goes astray with many repetitions #2287)
• median of the sum of runtimes per instance
• sum of runtimes
• sum of runtimes per success, aka ERT

The first three are monotonously increasing.

As we preferably make no distinction between instances and restarts, the second does not give a preferable analytical viewpoint.

For one success, the last two are the same. This suggests to display the sum of runtimes divided by max(1, number of successes)?