autocnn_unsup

Matlab scripts implementing the model from Recursive Autoconvolution for Unsupervised Learning of Convolutional Neural Networks accepted to IJCNN-2017. There is the previous version of this paper and a corresponding reference sheet. There is also simple Python code to learn filters with recursive autoconvolution and k-means.

Scripts for MNIST (autocnn_mnist.m), CIFAR-10 (CIFAR-100) (autocnn_cifar.m) and STL-10 (autocnn_stl10.m) are available.

To reproduce results from the paper, run scripts from the experiments folder.

Example of running

opts.matconvnet = 'your_path/matconvnet';
opts.vlfeat = 'your_path/vlfeat/toolbox/mex/mexa64';
opts.gtsvm = 'your_path/gtsvm/mex';
opts.n_folds = 10;
opts.n_train = 4000;
opts.PCA_dim = 1500;
opts.arch = '1024c5-3p-conv0_3__32g-32ch-256c5-3p-conv0_2__32g-256ch-1024c5-3p-conv0_2';
autocnn_cifar(opts, 'learning_method', {'kmeans','pca','pca'}, 'augment', true)

This script should lead to an average accuracy of about 79.4% on CIFAR-10 (400).

Requirements

For faster filter learning it's recommended to use VLFeat, and for faster forward pass - MatConvNet. Although, in the scripts we try to make it possible to choose between built-in Matlab and third party implementations.

For classification it's required to install either GTSVM, LIBLINEAR or LIBSVM. Compared to LIBSVM, GTSVM is much faster (because of GPU) and implements a one-vs-all SVM classifier (which is usually better for datasets like CIFAR-10 and STL-10). LIBLINEAR shows worse performance compared to the RBF kernel available both in GTSVM and LIBSVM. If neither of these is available, the code will use Matlab's LDA.

Learning methods

Currently, the supported unsupervised learning methods are k-means, convolutional k-means, k-medoids, GMM, PCA, ICA and ISA. We use VLFeat's k-means to obtain our results.

Testing environment

• Ubuntu 16.04 LTS
• Matlab R2015b
• CUDA 7.5 (installed via apt-get install nvidia-cuda-toolkit)
• MatConvNet
• cuDNN-v5
• VLFeat
• GTSVM
• 64GB RAM
• NVIDIA GTX 980 Ti
• Xeon CPU E5-2620 v3 @ 2.40GHz

Results

• The model is purely unsupervised, i.e., label information is not used to train filters.
• flip - flipping (horizontal reflection, mirroring) is applied both for training and test samples.
• augment - taking random crops, flipping, rotation and scaling.

A convolutional neural network (CNN) trained layer wise using unsupervised learning methods and recursive autoconvolution is abbreviated as AutoCNN.

MNIST

Model	MNIST (100)	MNIST
AutoCNN-S1-128 + LinearSVM	2.45 ± 0.10	0.69
AutoCNN-S2 + LinearSVM	1.75 ± 0.10	0.39

AutoCNN-S1-128: 128c11-4p-conv1_3

AutoCNN-S2: 128c7-5p-3s-conv1_3__1g-128ch-1024c5-3p-conv0_2

CIFAR-10 and CIFAR-100

Test accuracy (%) on CIFAR-10 (400) with 400 labeled images per class and using all (50k) training data of CIFAR-10 and CIFAR-100.

Model	CIFAR-10 (400)	CIFAR-10	CIFAR-100
AutoCNN-L + flip + LinearSVM	77.6 ± 0.3	84.4	-
AutoCNN-L32 + RBFSVM	76.4 ± 0.4	85.4	63.9
AutoCNN-L32 + flip + RBFSVM	79.4 ± 0.3	87.9	67.8

AutoCNN-L: 256c5-3p-conv0_3__1g-256ch-1024c5-3p-conv0_3__1g-1024ch-2048c5-3p-conv0_2

AutoCNN-L32: 1024c5-3p-conv0_3__32g-32ch-256c5-3p-conv0_2__32g-256ch-1024c5-3p-conv0_2

STL-10

Average test accuracy (%) on STL-10 using 10 predefined folds.

Model	STL-10
AutoCNN-L + augment + LinearSVM	73.1 ± 0.5
AutoCNN-L32 + RBFSVM	68.7 ± 0.5
AutoCNN-L32 + augment + RBFSVM	74.5 ± 0.6

AutoCNN-L: 256c7-4p-conv0_3__1g-256ch-1024c5-4p-conv0_3__1g-1024ch-2048c5-3p-conv0_2

AutoCNN-L32: 1024c7-5p-conv0_3__32g-32ch-256c5-4p-conv0_2__32g-256ch-1024c5-3p-conv0_2

AutoCNN-L32 + augment: 1024c7-4p-conv0_3__32g-32ch-256c5-4p-conv0_2__32g-256ch-1024c5-3p-conv0_2

Citation

@inproceedings{knyazev2017autoconvolution,
  title={Recursive Autoconvolution for Unsupervised Learning of Convolutional Neural Networks},
  author={Knyazev, Boris and Barth, Erhardt and Martinetz, Thomas},
  booktitle={International Joint Conference on Neural Networks},
  year={2017}
}