Support for nested parameters/parameterizing objects that can't be called by name.

[ben-arnao]

In a few different scenarios you'd want to have nested parameters, but sometimes a parameter isn't just a numerical/string value but instead a layer or object. An obvious example would be an optimizer with a learn rate. You need to call the learn rate inside of the optimizer initialization function.

So while you can have an optimizer as a hyper parameter like in the default example...

def build_fn(input_shape):
    model = Sequential([
        Dense(Integer(50, 150), input_shape=input_shape, activation='relu'),
        Dropout(Real(0.2, 0.7)),
        Dense(1, activation=Categorical(['sigmoid', 'softmax']))
    ])
    model.compile(
        optimizer=Categorical(['adam', 'rmsprop', 'sgd', 'adadelta']),
        loss='binary_crossentropy', metrics=['accuracy']
    )
    return model

And you can have learn rate as a param for a static optimizer

def build_fn(input_shape):
    model = Sequential([
        Dense(Integer(50, 150), input_shape=input_shape, activation='relu'),
        Dropout(Real(0.2, 0.7)),
        Dense(1, activation=Categorical(['sigmoid', 'softmax']))
    ])
    model.compile(
        optimizer=Adam(lr=Real(0.001, 0.1)),
        loss='binary_crossentropy', metrics=['accuracy']
    )
    return model

I'm not sure how you can have both as parameters simultaneously.

What i have done in my own custom setup to get around a similar issue is to have a wrapper function that returns a optimizer. This way i can return optimizers that can't be called by name (Ie. Adamax or Nadam). With this same approach i think i could also have learn rate in the wrapper function as well. So i could do something like to get the functionality i want.

get_custom_optimizer(Categorical(['adam', 'nadam')], Real(0.001, 0.01))

Which should just return an Adam or Nadam optimizer, with a random learn rate.

The problem is i don't think you can call any non-native function inside your build_fn...

For example:

def get_opt():
    return Adamax()

def build_fn(input_shape):
    model = Sequential([
        Dense(Integer(50, 150), input_shape=input_shape, activation='relu'),
        Dropout(Real(0.2, 0.7)),
        Dense(1, activation=Categorical(['sigmoid', 'softmax']))
    ])
    model.compile(
        optimizer=get_opt(),
        loss='binary_crossentropy', metrics=['accuracy']
    )
    return model

Results in an error
NameError: name 'get_opt' is not defined

[HunterMcGushion]

Thanks for raising this!

You’re absolutely right. Currently, we don’t support optimizing both a Keras optimizer object and its parameters at the same time. As you say, optimizing nested parameters in general currently isn’t supported. However the Keras optimizers are specifically problematic because they each have different parameters.

While all optimizers do have an lr kwarg, some have other kwargs that different optimizers don’t have. For example, an attempt to search through optimizers “sgd” and “rmsprop”, as well as momentum gets weird because RMSprop doesn’t have a momentum kwarg. Furthermore, many optimizers have different default lr values.

As you mentioned, this could probably be addressed by adding support for nested/conditional hyperparameter optimization, but I still think it gets messy rather quickly in the case of Keras optimizers. This messiness is compounded when we consider that HH is going to automatically read all of our old Experiments and try to match them with the new search space to jump-start optimization. All the old Experiment results, of course, will also have the default values for hyperparameters that weren’t explicitly given.

All that said, I would very much like to support nested/conditional dimensions…

Could you please clarify what you mean by,

This way i can return optimizers that can’t be called by name (Ie. Adamax or Nadam)

If you’re referring to using Categorical on the string names of optimizers, then “adamax” and “nadam” do work; however, I think I may have misunderstood.

…

The problem is i don’t think you can call any non-native function inside your build_fn…

That’s correct. I won’t bore you with all the technical details, but HH actually rewrites the Keras build_fn its given before evaluating it, and this currently leaves out any non-native or non-Keras stuff, which is the reason for the NameError you’re getting.

Doing this may seem unnecessarily complicated, but the reason for it is to enable providing search dimensions directly to Keras layers. Otherwise it wouldn’t be possible to do something like Dense(Integer(50, 150)) because Dense is expecting a native Python int, not an Integer instance, as its first argument.

If you have any ideas or would actually like some more technical details, I’d be more than happy to discuss it further, but that’s the “short” answer to why you’re getting that NameError.

Your wrapper function approach does seem really interesting, though, and I’m trying to figure out how we could integrate it into HH to support nested/conditional dimensions.

[ben-arnao]

If you’re referring to using Categorical on the string names of optimizers, then “adamax” and “nadam” do work; however, I think I may have misunderstood.

adamax and nadam are callable by name? I wasn't aware. Either way i think we'd have the same problem when using custom optimizers like adabound, that really have to be defined in a wrapper function.

Your wrapper function approach does seem really interesting, though, and I’m trying to figure out how we could integrate it into HH to support nested/conditional dimensions.

Maybe it is possible to allow a list of methods as params in the build_fn function? Then if we annotated our wrappers HH could possibly supply the appropriate function by name. This way we can supply our own wrapper functions. I'm probably over simplifying things...