Scalable-L2O

Implementation of ECCV 2022 paper: Scalable Learning to Optimize: A Learned Optimizer Can Train Big Models.

Environment

We recommend using Anaconda to manage the virtual environment.

conda env create -f environment.yaml
conda activate sl2o

Experiments

CNNs

Subspaces

We provide pre-generated subspaces for models in this link. Optionally, one can generate the subspaces by themselves. Please refer to the subspaces directory for more details.

Meta-Training

ResNet-18 (CIFAR-10)

python -u src/resnet18_ft_de.py --max_epoch 20 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --meta_train_eval_epoch 2 

ResNet-18 (CIFAR-100)

python -u src/resnet18_ft_de.py --max_epoch 20 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --meta_train_eval_epoch 2 --dataset CIFAR100

ResNet8 (CIFAR-10)

python -u src/resnet8_ft_de.py --max_epoch 20 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --meta_train_eval_epoch 2 

ResNet20 (CIFAR-10)

python -u src/resnet20_ft_de.py --max_epoch 20 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --meta_train_eval_epoch 2

Meta-Testing

ResNet-8 (CIFAR-10)

python -u src/resnet8_eval_de.py --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --max_epoch 100

ResNet-20 (CIFAR-10)

python -u src/resnet20_eval_de.py --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 128 --unroll 10 --max_epoch 100

VITs

Meta-Training

python -u src/vit_ft.py --max_epoch 20 --lora_dim 16 --lora_alpha 32 --lora_dropout 0.1 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 5 --training_steps 1000 --batch-size 64 --unroll 10 --random_seed 1 --name cifar10-100_500 --dataset cifar10 --model_type ViT-B_16 --pretrained_dir checkpoint/ViT-B_16.npz --meta_train_eval_epoch 2 

Meta-Testing

python -u src/vit_ft_eval.py --max_epoch 20 --lora_dim 16 --lora_alpha 32 --lora_dropout 0.1 --eval_interval 2000 --log_interval 100 --hidden_sz 8 --scale 1e-4 --log_interval 10 --training_steps 1000 --batch-size 64 --unroll 10 --random_seed 1 --name cifar10-100_500 --dataset cifar10 --model_type ViT-B_16 --pretrained_dir checkpoint/ViT-B_16.npz --random_seed 1 --eval_interval 391