Trying to benchmark all the optimizers used in deep learning

Network

conv1 - 64 - 55 - 11
maxpool2 - 33, 22
local_response_normalization
conv2 - 64 - 55 - 11
maxpool2 - 33, 22
local_response_normalization
FC - 384
FC - 192
FC - Final (Mnist , CIFAR10 - 10, CIFAR100 - 100)

Optimizers to test

• adadelta
• adagrad
• adam
• ftrl
• momentum
• rmsprop
• sgd

Things to check

• learning_rate behaviour with-in each optimizer
• time taken to run each optimizer
• hyper-parameters to tune (check for one)
• maximum training, testing and validation accuracy achieved
• graphs

Datasets

• MNIST
• CIFAR10
• CIFAR100

The models are trained with batch_size: 32 epochs: 100

train, validation and test accuracies are obtained after the 100th epoch.

#Experiments

1. Tensorflow default values:

• We have used a learning_rate of 0.001 for all the optimizers keeping all other params constant
• Momentum=0.9 in case of momentum optimizer

MNIST Dataset

optimizer	train_accuracy	Validation_Accuracy	Test Accuracy
adadelta	00.00%	00.00%	00.00%
adagrad	00.00%	00.00%	00.00%
adam	00.00%	00.00%	00.00%
ftrl	00.00%	00.00%	00.00%
momentum	00.00%	00.00%	00.00%
rmsprop	00.00%	00.00%	00.00%
sgd	00.00%	00.00%	00.00%

CIFAR10 Dataset

optimizer	train_accuracy	Validation_Accuracy	Test Accuracy
adadelta	00.00%	00.00%	00.00%
adagrad	00.00%	00.00%	00.00%
adam	00.00%	00.00%	00.00%
ftrl	00.00%	00.00%	00.00%
momentum	00.00%	00.00%	00.00%
rmsprop	00.00%	00.00%	00.00%
sgd	00.00%	00.00%	00.00%

CIFAR100 Dataset

optimizer	train_accuracy	Validation_Accuracy	Test Accuracy
adadelta	00.00%	00.00%	00.00%
adagrad	00.00%	00.00%	00.00%
adam	00.00%	00.00%	00.00%
ftrl	00.00%	00.00%	00.00%
momentum	00.00%	00.00%	00.00%
rmsprop	00.00%	00.00%	00.00%
sgd	00.00%	00.00%	00.00%