CHIP: CHannel Independence-based Pruning for Compact Neural Networks

Code for our NeurIPS 2021 paper: (CHIP: CHannel Independence-based Pruning for Compact Neural Networks).

Usage

Generate Feature Maps.

1. CIFAR-10

python calculate_feature_maps.py \
--arch resnet_56 \
--dataset cifar10 \
--data_dir ./data \
--pretrain_dir ./pretrained_models/resnet_56.pt \
--gpu 0

2. ImageNet

python calculate_feature_maps.py \
--arch resnet_50 \
--dataset imagenet \
--data_dir /raid/data/imagenet \
--pretrain_dir ./pretrained_models/resnet50.pth \
--gpu 0

Generate Channel Independence (CI).

This procedure is time-consuming, please be patient.

1. CIFAR-10

python calculate_ci.py \
--arch resnet_56 \
--repeat 5 \
--num_layers 55

2. ImageNet

python calculate_ci.py \
--arch resnet_50 \
--repeat 5 \
--num_layers 53

Prune and Fine-tune Models.

1. CIFAR-10

python prune_finetune_cifar.py \
--data_dir ./data \
--result_dir ./result/resnet_56/1 \
--arch resnet_56 \
--ci_dir ./CI_resnet_56 \
--batch_size 256 \
--epochs 400 \
--lr_type cos \
--learning_rate 0.01 \
--momentum 0.9 \
--weight_decay 0.005 \
--pretrain_dir ./pretrained_models/resnet_56.pt \
--sparsity [0.]+[0.4]*2+[0.5]*9+[0.6]*9+[0.7]*9 \
--gpu 0 

2. ImageNet

python prune_finetune_imagenet.py \
--data_dir /raid/data/imagenet \
--result_dir ./result/resnet_50/1 \
--arch resnet_50 \
--ci_dir ./CI_resnet_50 \
--batch_size 256 \
--epochs 180 \
--lr_type cos \
--learning_rate 0.01 \
--momentum 0.99 \
--label_smooth 0.1 \
--weight_decay 0.0001 \
--pretrain_dir ./pretrained_models/resnet50.pth \
--sparsity [0.]+[0.1]*3+[0.35]*16 \
--gpu 0

Pre-trained Models

• Pre-trained Models
  • CIFAR-10: VGG-16_BN, ResNet-56, ResNet-110.
  • ImageNet: ResNet-50.

Results

We release our training logs of ResNet-56/110 model on CIFAR-10 for more epochs which can achieve better results than paper. We release our training logs of ResNet-50 model on ImageNet. Training logs can be found at link. Some results are better than papers.

CIFAR-10

Model	# of Params (Reduction)	Flops (Reduction)	Top-1 Accuracy	Sparsity Setting
ResNet-56	0.85M(0.0%)	125.49M(0.0%)	93.26%	N/A
ResNet-56	0.48M(42.8%)	65.94M(47.4%)	94.16%	[0.]+[0.15]*2+[0.4]*27
ResNet-56	0.24M(70.0%)	34.78M(74.1%)	92.43%	[0.]+[0.4]*2+[0.5]*9+[0.6]*9+[0.7]*9
ResNet-110	1.72M(0.0%)	252.89M(0.0%)	93.50%	N/A
ResNet-110	1.04M(39.1%)	140.54M(44.4%)	94.50%	[0.]+[0.2]*2+[0.3]*18+[0.35]*36
ResNet-110	0.89M(48.3%)	121.09M(52.1%)	94.44%	[0.]+[0.22]*2+[0.35]*18+[0.45]*36
ResNet-110	0.54M(68.3%)	71.69M(71.6%)	93.23%	[0.]+[0.4]*2+[0.5]*18+[0.65]*36
VGG-16-BN	14.98M(0.0%)	313.73M(0.0%)	93.96%	N/A
VGG-16-BN	2.76M(81.6%)	131.17M(58.1%)	93.86%	[0.21]*7+[0.75]*5
VGG-16-BN	2.50M(83.3%)	104.78M(66.6%)	93.72%	[0.3]*7+[0.75]*5
VGG-16-BN	1.90M(87.3%)	66.95M(78.6%)	93.18%	[0.45]*7+[0.78]*5

ImageNet

Model	# of Params (Reduction)	Flops (Reduction)	Top-1 Accuracy	Top-5 Accuracy	Sparsity Setting
ResNet-50	25.55M(0.0%)	4.11B(0.0%)	76.15%	92.87%	N/A
ResNet-50	15.09M(40.8%)	2.26B(44.8%)	76.41%	93.06%	[0.]+[0.1]*3+[0.35]*16
ResNet-50	14.28M(44.2%)	2.19B(48.7%)	76.35%	93.05%	[0.]+[0.12]*3+[0.38]*16
ResNet-50	11.05M(56.7%)	1.52B(62.8%)	75.26%	92.53%	[0.]+[0.25]*3+[0.5]*16
ResNet-50	8.02M(68.6%)	0.95B(76.7%)	73.30%	91.48%	[0.]+[0.5]*3+[0.6]*16

Others

Codes are based on link.

Since I rearranged my original codes for simplicity, please feel free to open an issue if something wrong happens when you run the codes. (Please forgive me for the late response and wait for me to respond to your problems in several days.)