HyperparameterOptimizationFacade does wrong initial_design for hyperparameters from type NormalInteger

[Priapos1004]

Description

I am currently trying to use HyperparameterOptimizationFacade for my RandomForestClassifier and defined a ConfigSpace.ConfigurationSpace for its hyperparameters. The problem is that the .get_initial_design() function is returning a not working configuration with "max_depth=0". This should not be possible with the used ConfigSpace.ConfigurationSpace. I tested it for other grids and it seems that there is a problem with handling NormalInteger type in the defined grid.

Steps/Code to Reproduce

from ConfigSpace import Categorical, ConfigurationSpace, Integer, Normal
from smac import HyperparameterOptimizationFacade, Scenario

grid = ConfigurationSpace(
            seed=42,
            space={
            "n_estimators": Integer("n_estimators", (10, 1000), log=True, default=100),
            "max_depth": Integer("max_depth", (3, 15), distribution=Normal(5, 3), default=5),
            "min_samples_split": Integer("min_samples_split", (2, 10), default=2),
            "min_samples_leaf": Integer("min_samples_leaf", (1, 4), default=1),
            "bootstrap": Categorical("bootstrap", [True, False], default=True),
            "criterion": Categorical("criterion", ["gini", "entropy"], default="gini"),
            })

scenario = Scenario(
    grid,
    n_trials=10,
    deterministic=True,
    walltime_limit=200,
)

initial_design = HyperparameterOptimizationFacade.get_initial_design(scenario, n_configs=5)
initial_design.select_configurations()

Expected Results

A valid configuration so that no error will occur in HyperparameterOptimizationFacade because of not possible parameter values.

Actual Results

Versions

smac: 2.0.1
ConfigSpace: 0.6.1

[dengdifan]

Hi,

Thanks for the information. This issue might be related to ConfigSpace: automl/ConfigSpace#302
Could you please raise an issue there?
As a temporary solution, you could also consider if Beta distribution fits your requirement: https://automl.github.io/ConfigSpace/main/api/hyperparameters.html#ConfigSpace.hyperparameters.BetaIntegerHyperparameter

[eddiebergman]

This is likely not related to the issue you posted @dengdifan

ConfigSpace seems to be behaving quite nicely from the example given:

from ConfigSpace import Categorical, ConfigurationSpace, Integer, Normal

grid = ConfigurationSpace(
    seed=42,
    space={
        "n_estimators": Integer("n_estimators", (10, 1000), log=True, default=100),
        "max_depth": Integer("max_depth", (3, 15), distribution=Normal(5, 3), default=5),
        "min_samples_split": Integer("min_samples_split", (2, 10), default=2),
        "min_samples_leaf": Integer("min_samples_leaf", (1, 4), default=1),
        "bootstrap": Categorical("bootstrap", [True, False], default=True),
        "criterion": Categorical("criterion", ["gini", "entropy"], default="gini"),
    },
)

max_depths_sampled = {
    grid.sample_configuration()["max_depth"] for i in range(10000)
}
print(sorted(max_depths_sampled))
# [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]

[eddiebergman]

Some further digging into SMAC, I'm assuming it's this line that's causing the issue:

SMAC3/smac/initial_design/abstract_initial_design.py

Lines 180 to 181 in e64e191

	if isinstance(param, IntegerHyperparameter):
	design[:, idx] = param._inverse_transform(param._transform(design[:, idx]))

I'm not exactly sure what's in the design param there but using _transform and then _inverse_transform is likely causing it to ignore the bounds somehow.

tldr; likely a ConfigSpace issue just a different one

[alexandertornede]

@eddiebergman Did you raise the issue above at ConfigSpace?