This repository is the official implementation of "Revisiting Orthogonality Regularization: A Study for Convolutional Neural Networks in Image Classification" for the image classification tasks, published at IEEE Access 2022 Journal (Paper Link).
Experiments in our paper can be reproduced as the following .sh files. To train a vanilla model, firstly run this command:
bash run_vanilla.sh
You can get other baseline models by changing --model argument. Also, you can also get the baseline model on CIFAR10 by changing --dataset arguments to cifar10, cifar100, imagenet. To give an orthogonality constraint, you can change --ortho=none argument such as srip, ort,ortho,sin_srip,isometry. You can find this in ./regularization/orthogonal_weight.py. Besides, you may further rescale the lambda value via the argument --lamb_list. --lamb_list contains four values, and they indicates the lambda for different layers: convolutionaly filter with kernel size 3, pointwise filter, depthwise filter, fully connected layer. --wd_ablation implies the switch of weight decay. Precisely, if you give other regularizations to certain layer, then weight decay will not be applied when --wd_ablation turns on. Furthermore, we add lots of options for experiments such as initialization (--init), optimization (--optim), scheduler (--lr_sch). Even, we add SOTA algorithms for image classification such as label smoothing, mix up, and cutmix.
Current code base is tested under following environment:
- Python 3.7.3
- PyTorch 1.6.0
- torchvision 0.7.0
- Numpy 1.19.2
- Pandas 1.2.0
- tqdm
You can reproduce all results in the paper with our code. All results have been described in our paper including Appendix. The results of our experiments are so numerous that it is difficult to post everything here. However, if you experiment several times by modifying the hyperparameter value in the .sh file, you will be able to reproduce all of our analysis.