surgery

Evaluating Efficient Performance Estimators of Neural Architectures

If you find this work/code helpful, please cite:

@inproceedings{
ning2021evaluating,
title={Evaluating Efficient Performance Estimators of Neural Architectures},
author={Xuefei Ning and Changcheng Tang and Wenshuo Li and Zixuan Zhou and Shuang Liang and Huazhong Yang and Yu Wang},
booktitle={Thirty-Fifth Conference on Neural Information Processing Systems},
year={2021}
}

For an overview of our work, we encourage the readers to also check the poster here.

Code

The implementation of the NAS components for the three benchmarks, including the search space, GT evaluator, weights manager, (optional) architecture embedder and so on, are under the sub-package aw_nas.btcs. Check the READMEs in the sub-directories: nb101, nb201, and nb301 for more instructions on how to run one-shot training, architecture rewards estimation (derive and eval-arch), and criteria evaluation of the estimation results.

The implementation of the non-topological search spaces ResNet/ResNeXt is in a plugin code snippet. Check the [nds][./nds] subdirectory for more instructions on how to run OS training/evaluation on NDS ResNet and ResNeXt.

There are many other assets (logs, configuration files, helper scripts) used in our paper, and the main assets can be found at this URL.

There are several helper scripts provided under this directory. They are

• run_supernet_training.py: Run multiple awnas search on multiple GPUs.
• run_derive_ckpts.py: Run multiple awnas derive (NB201) or awnas eval-arch (NB301/NB101) on multiple GPUs.
• run_oneshot_batch.py : Run awnas eval-arch, and return the rewards as a list, in which each item is the reward in a valdiation batch (This script add a configuration cfg["objective_cfg"]["aggregate_as_list"] = True, which means to collect rewards of multiple batches into a list instead of averaging them to get a scalar.).
• run_zero_shot.py: Run zero-shot estimations. By default, multiple random initialization seeds would be used (20, 2020, 202020). Optionally, one can specificy --evaluators to calculate the zero-shot indicators on pretrained one-shot supernets.
• evaluation.py: Evaluate the derive (NB201) or eval-arch (NB301/101) results, and dump the corresponding criteria (i.e., One-shot average, Kendall's Tau, SpearmanR, P@top/bottom Ks, B/WR@K, etc.) into a pickle file with the _statistics.pkl suffix.

Please check the help messages as well as the codes to see how to run these helper scripts.

NAS-Bench-201

See instructions under nb201/.

NAS-Bench-301

See instructions under nb301/.

NAS-Bench-1shot1-Sub3

See instructions under nb101/.

NDS ResNet and ResNeXt (Non-topological search spaces)

See instructions under nds/.

Zero-shot Estimators (ZSEs)

We appreciate the authors of [Zero-Cost Proxies for Lightweight NAS, ICLR2021] for their efforts in providing the research codes. In order to evaluate ZSEs, we integrate their lib into our framework in aw_nas/objective/zerocost.py. Thus their lib should be installed for evaluating ZSEs. To evaluate relu_logdet and relu scores, one should use our fork of their lib.

The zeroshot configuration files on different search spaces are nb101/zeroshot.yaml, nb201/zeroshot.yaml, nb301/zeroshot.yaml, nds/resnet_zeroshot.yaml, and resnexta_zeroshot.yaml.

A Final Note: If you find the instructions in this repository too intense (we hope not), there are some scripts to run convenient example runs. These scripts & instructions are in our anonymized supplementary file, and we upload it into google drive here for your reference.

This zip file is not small (97M), since there are also checkpoints and logs in it. The code & instructions in this zip file is (maybe) easier to follow and self-contained. That is to say, if you use this anonymized supplementary code, you do not need this repository.