orion.core.worker.algo_wrappers.insist_suggest::Unable to sample a new trial from the algorithm, even after 100 attempts! Returning an empty list.

[cometta]

may i know error

2023-06-26 01:10:04,933::WARNING::orion.core.worker.algo_wrappers.insist_suggest::Unable to sample a new trial from the algorithm, even after 100 attempts! Returning an empty list.

when using hyperdad, i use default max_trials, which is very huge number, is the above error due to I use the default max_trials instead of setting small number?

[bouthilx]

It may be due to the search space which is too small, so the algorithm keep sample duplicates. Can you please share your configuration, with search space and algorithm configuration?

[cometta]

experiment:
    algorithm:
        hyperband:
            seed: 1234
            repetitions: 1
evc:
    enable: True
    algorithm_change: True
    auto_resolution: True
    cli_change_type: break
    code_change_type: break
    config_change_type: break
    orion_version_change: False
    ignore_code_changes: False
    manual_resolution: False
    non_monitored_arguments: []

command is below, i masked real option names

orion hunt -n find_best_parameters_hyperparameter --config orion_config.yaml python do_something.py --optimizer~"choices(['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L','M','N','O', 'P', 'Q', 'R', 'S'])" --noise_factors~"choices(['option-1','option-2'])" --noise_amplifier~"choices(['one', 'two', 'three', 'four'])" --extrapolator~"choices(['one', 'two', 'three','four'])" --maxiter~"fidelity(1, 100,base=2)" 

[bouthilx]

This gives only 608 different possible configurations. Hyperband won't help much compared to a basic random search. If some of these variables are integers or continuous values I would recommend using uniform() or loguniform() instead of categorical values.

[cometta]

noted. this happend for TPE, ASHA, evolution as well. Does that mean all other algo has similar issue except random search?

[bouthilx]

TPE can have difficulty sampling all possible configurations as it focuses on the best combinations, leading to this problem. ASHA, Hyperband and EvolutionaryES will fail because they need to sample enough configurations to start doing promotions (increase number of epochs). Random search simply samples until it reaches max_trials or exhausted all possible configurations.