This repository provides the implementation of the method proposed in our paper Pruning Deep Neural Networks using Partial Least Squares. The code in this repository is able to prune simple CNN architectures (i.e., VGG-based). To prune more sophisticated networks (i.e., ResNets), you can employ our method to identify potential filters to be removed and use Keras-Surgeon to rebuild the network. The figure below illustrates our main results by pruning ResNet56 and ResNet110 on the CIFAR-10 dataset.
- Scikit-learn
- Keras (Recommended version 2.1.2)
- Tensorflow (Recommended version 1.3.0 or 1.9)
- Python 3
main.py provides an example of our pruning approach. In this example, we prune a simple convolutional neural network. It should be mentioned that the network of this example is not the network used in our paper. Instead, we prefer to use this simple network due to the computational cost.
Our method takes two parameters:
- Number of pruning iterations (see line 138 in main.py)
- Percentage of filters to be removed in each iteration (see line 139 in main.py)
- Number of components of Partial Least Squares (see line 141 in main.py)
- Filter representation (see line 192 in main.py). The options are: 'max' and 'avg'. In addition, you can customize a pooling operation (i.e., max-pooling 2x2) to represent the filters (see line 255 in main.py)
Tables below show the comparison between our method with existing pruning methods. Negative values in accuracy denote improvement regarding the original network. Please check our paper for more detailed results. We provide the models and weights to reproduce these results in models/CIFAR10.
ResNet56 on Cifar-10
| Method | FLOPs ↓ (%) | Accuracy ↓ (percentage points) |
|---|---|---|
| Yu et al. | 43.61 | 0.03 |
| He et al. | 50.00 | 0.90 |
| Ours(it=1) | 7.09 | -0.60 |
| Ours(it=5) | 35.23 | -0.90 |
| Ours(it=8) | 52.56 | -0.62 |
ResNet50 on ImageNet
| Method | FLOPs ↓ (%) | Accuracy ↓ (percentage points) |
|---|---|---|
| Liu et al. | 36.70 | 1.01 |
| He et al. | 41.80 | 8.27 |
| He et al. | 53.50 | 0.55 |
| Ours (it=1) | 6.13 | -1.92 |
| Ours (it=5) | 27.45 | -0.31 |
| Ours (it=10) | 44.50 | 1.01 |
Please cite our paper in your publications if it helps your research.
@inproceedings{Jordao:2019,
author = {Artur Jordao,
Ricardo Kloss,
Fernando Yamada and
William Robson Schwartz},
title = {Pruning Deep Neural Networks using Partial Least Squares},
booktitle = {British Machine Vision Conference (BMVC) Workshops: Embedded AI for Real-Time Machine Vision},
}
@article{Jordao::2020,
author = {Artur Jordao,
Fernando Yamada and
William Robson Schwartz},
title = {Deep network compression based on Partial Least Squares},
journal = {Neurocomputing},
year = {2020},
}We would like to thank Maiko Lie for the coffees and talks.