DARTS for RNN with fastai

Language model on Penn Treebank using Differentiable Architecture Search (DARTS) and fastai library.
Blog post.
Based on DARTS: Differentiable Architecture Search by Hanxiao Liu, Karen Simonyan, Yiming Yang.
Check out the original implementation.

Requirements

fastai 1.0.52.dev0 (latest as of 10th April 2019), PyTorch 1.0.

Instructions

1. Run databunch_nb.ipynb to create databunch
2. Run train_search_nb.ipynb to search for genotype. ~5 hours on 1 v100 gpu for 1 run.
   RNN search is sensitive to initialization so there should be several runs with different seed
3. Train that genotype from scratch on train_nb.ipynb. ~1.5 days for 1600 epochs.
4. Test a model using test_nb.ipynb

Pretrained model

Pretrained model of DARTS_V1 genotype after 600 epochs darts_V1.pth.
Place the file at data/models and run test_nb.ipynb. Loss ~4.22, 68.0 perplexity.
Caveat: I haven't been able to get ~58.0 test perplexity like the original implementation.

fastai dev version installation

git clone https://github.com/fastai/fastai
cd fastai
tools/run-after-git-clone
pip install -e ".[dev]"
git pull