| title | tags | authors | affiliations | date | bibliography | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Hyperas: Simple Hyperparameter Tuning for Keras Models |
|
|
|
19 November 2021 |
paper.bib |
Hyperas is an extension of Keras [@chollet2015keras], which allows you to run hyperparameter optimization of your models using Hyperopt [@bergstra2012hyperopt]. It was built to enable fast experimentation cycles for researchers and software developers. With hyperas, you can set up your Keras models as you're used to and specify your hyperparameter search spaces in a convenient way, following the design principles suggested by the Jinja project [@jinja2008].
With hyperas, researchers can use the full power of hyperopt without sacrificing experimentation speed. Its documentation is hosted on GitHub and comes with suite of [examples]https://github.com/maxpumperla/hyperas/tree/master/examples) to get users started.
Hyperas is in active use in the Python community and still has thousands of weekly downloads, which shows a clear need for this experimentation library. Over the years, hyperas has been used and cited by research papers, mostly by referring to Github. Researchers that want to focus on their deep learning model definitions don't get bogged down by maintaining separate hyperparameter search spaces and configurations and can leverage hyperas to speed up their experiments. After hyperas has been published, tools like Optuna [@akiba2019optuna] have adopted a similar approach to hyperparameter tuning. KerasTuner [@omalley2019kerastuner] is officially supported by Keras itself, but does not have the same variety of hyperparameter search algorithms as hyperas.
Hyperas uses a Jinja-style template language to define search spaces implicitly in Keras model specifications.
Essentially, regular configuration values in a Keras layer, such as Dropout(0.2) get replaced by a suitable distribution like Dropout({{uniform(0, 1)}}).
To define a hyperas model, you proceed in two steps.
First, you set up a function that returns the data you want to train on, which could include features and labels for training, validation and test sets.
Schematically this would look as follows:
def data():
# Load your data here
return x_train, y_train, x_test, y_testNext, you have to specify a function that takes your data as input arguments, defines a Keras model with hyperas template handles ({{}}), fits the model to your data and returns a dictionary that has to at least contain a loss value to be minimized by hyperopt, e.g. validation loss or the negative of test accuracy, and the hyperopt status of the experiment.
from hyperas.distributions import uniform
from hyperopt import STATUS_OK
def create_model(x_train, y_train, x_test, y_test):
model = Sequential()
model.add(Dense(512, input_shape=(784,)))
model.add(Activation('relu'))
model.add(Dropout({{uniform(0, 1)}}))
# ... add more layers
model.add(Dense(10))
model.add(Activation('softmax'))
# fit model
model.fit(x_train, y_train, ...)
# evaluate model and return loss
score = model.evaluate(x_test, y_test, verbose=0)
accuracy = score[1]
return {'loss': -accuracy, 'status': STATUS_OK, 'model': model}Lastly, you simply prompt the optim module of hyperas to minimize your model loss defined in create_function, using data, with a hyperparameter optimization algorithm like TPE or any other algorithm supported by hyperopt [@pmlr-v28-bergstra13].
from hyperas import optim
from hyperopt import Trials, STATUS_OK, tpe
best_run = optim.minimize(model=create_model,
data=data,
algo=tpe.suggest,
max_evals=10,
trials=Trials())Furthermore, note that hyperas can run hyperparameter tuning in parrallel, using hyperopt's distributed MongoDB backend.
We would like to thank all the open-source contributors that helped making hyperas what it is today.
It's a great honor to see your software continually used by the community.