Neural Optimizer Search - Optimizer_1

PyTorch implementation of Neural Optimizer Search's Optimizer_1

Usage

Import Optimizer_1 like any torch.optim Optimizer:

from optimizer_1 import Optimizer_1

optimizer = Optimizer_1(model.parameters(), lr=1e-3, momentum=0.99)
loss.backward()
optimizer.step()

---

Results

---

Own results

MNIST (PyTorch/examples)

Optimizer_1 Test set: Average loss: 0.0525, Accuracy: 9820/10000 (98%)

SGD with Momentum Test set: Average loss: 0.0542, Accuracy: 9825/10000 (98%)

VAE (PyTorch/examples)

Optimizer_1 Test set: Average loss: 111.2389

Adam Test set: Average loss: 105.5995