Anyone interested in helping tweak skopt's performance in a benchmarking exercise?

[microprediction]

Hello,

I am wondering if someone with more experience than me is interested in helping tweak skopt's performance in a benchmarking exercise I am running (in the open) for my company.

As you can see from the leaderboard skopt is beating a whole slew of the usual suspects. Nice job!

However, I'd like to include a few more sub-choices in gp_minimize and wondering if anyone has favourite choices of parameters. My use of skopt is here. I'm thus far trying the following variations:

'lcbexplore':{"acq_func": "LCB", "kappa":3.0},
'lcb':{"acq_func":"LCB","kappa":1.96},
'lcbexploit':{"acq_func":"LCB","kappa":1.0},
'pi':{"acq_func":"pi"},
'xi02':{"xi":0.02},
'default':{},
'sampling':{'acq_optimizer':'sampling'},
'lbfgs':{'acq_optimizer':'lbfgs'},
'noisy':{'noise':'gaussian'},
'calm':{'noise':1e-8},
'':{}

Pls let me know if something else should be tried.

Peter

[kernc]

The leaderboard link 404s.

Wouldn't better-performing combinations of parameters depend always solely on the distribution of the underlying data that is being fitted (i.e. no free lunch)?

Why not just do grid search over all accepted parameters/values?

[microprediction]

Thanks, I fixed the leaderboard URL. If you just want one of them, https://github.com/microprediction/optimizer-elo-ratings/tree/main/results/leaderboards/overall

Grid search could be included ... which would probably answer your question. Although remember that these optimizers are being challenged in quite high dimensions sometimes, so it isn't entirely clear what constitutes a grid search.

[kernc]

I meant grid search over viable parameters to gp_minimize(). (Or even a more informed, skopt-based search (over viable parameters to gp_minimize()) — go full meta. 😅)

[microprediction]

I'm with you now. Just a newbie so not sure how to describe the valid params.

[kernc]

I guess just from the documentation? E.g. kappa= is a float parameter, so you can "grid" it as np.linspace(0, 4, 9) or "skopt-space" it as Real(0, 4) ...