[Question] How to deal with high variance in the observation?

[showgood163]

Hello,

I have an optimization task on the hyperparameters of a model with following constraints:

1. The hparams are mostly int-type ranged parameters and it's impossible to try out every combination because the search space is very large.
2. The metric is kind of noisy (high variance).
3. The time consumption of running one evaluation is high.

Is there a way to deal with high variance in the observation? For now, I can only come up with:

1. For an arm, run the evaluation k times and report the mean and sem to the BO model
2. One evaluation for one arm and when the BO model suggests an existed trial, run another evaluation and update the mean and sem accordingly (which relates to [Sobol fallback needed] Repeated trials in experiment (and numerical errors they sometimes cause: RuntimeError: cholesky_cpu: U(63,63) is zero, singular U.) #228).

My questions are:

1. Which way above is better?
2. Is there a better way to deal with this kind of situation?

Thanks!

[Balandat]

Hi, to answer your questions:

1. Is depends. Do you expect the noise level to be reasonably homoskedastic (not to vary too much across the search space)? If so, then approach 2. that infers the noise level from individual observations is quite reasonable and possibly more data-efficient (in this case you would not pass an SEM, just the values of the repeated evaluations). If you expect the noise level to show significant heteroskedasticity (vary wildly across the search space) then solution 1. may be more appropriate.
2. I think this is right the way to go within Ax at this point. If one wanted to get fancy with the modeling one could try to fit a model that works with individual data points but infers a heteroskedastic noise model (there is work on this in BoTorch in this PR and we have some draft diff for hooking it to Ax, but it would need to be cleaned up and more thoroughly tested before we can merge it in).

Hope this helps.

[lena-kashtelyan]

@showgood163, did @Balandat answer your question? Can this issue be closed?

[showgood163]

@lena-kashtelyan
Sorry for the really late reply. I'll try the method as @Balandat told.