pytorch iCNN

Reconstructing image/movie from their features in arbitrary CNN model written in pytorch

	Image1	Image2
Original images
Reconstructed from feature[0]
Reconstructed from feature[2]
Reconstructed from features[10]
Reconstructed from features[12]
Reconstructed from features[21]
Reconstructed from classifier[0]

These images are reconstuctred from the activation (features) of layers in VGG16 trained on ImageNet dataset. You can reconstruct arbitorary images/videos from their features such as AlexNet, ResNet, DenseNet, and other architecture written in pytorch.

Description

The basic idea of the algorithm (Mahendran A and Vedaldi A (2015). Understanding deep image representations by inverting them. https://arxiv.org/abs/1412.0035) is that the image is reconstructed such that the CNN features of the reconstructed image are close to those of the target image. The reconstruction is solved by gradient based optimization algorithm. The optimization starts with a random initial image, inputs the initial image to the CNN model, calculates the error in feature space of the CNN, back-propagates the error to the image layer, and then updates the image.

Requirements

• Python 3.6 or later
• Numpy 1.19.0
• Scipy 1.5.1
• PIL 7.2.0
• Pytorch 1.5.1
• Torchvison 0.6.1
• Opencv 4.3.0 (if you want to only generate images, you don't need it)

Usage

The example of reconstructing image is at example/icnn_shortest_demo.ipynb.

Image Gallery

Heres are the example results of a various network.

#####resnet

Original image
Reconstructed from conv1
Reconstructed from layer1[0].conv2
Reconstructed from layer3[1].conv1

densenet

Original image
Reconstructed from features.denseblock1.denselayer1.conv1
Reconstructed from features.denseblock1.denselayer4.conv1
Reconstructed from features.denseblock3.denselayer24.conv1

This code also optimized multi layers output.

resnet

Original image
Reconstructed from first 2 layers
Reconstructed from first 3 layers
Reconstructed from 5 layers

Version

version 0.5 #released at 2020/07/13

version 0.8 #updated to show various DNN model at 2020/07/14

Copyright and license

The codes in this repository are based on Inverting CNN (iCNN): image reconstruction from CNN features(https://github.com/KamitaniLab/icnn) which is written for "Caffe" and Python2. These scripts are released under the MIT license.

Author

Ken Shirakawa

Ph.D. student at Kamitani lab, Kyoto University (http://kamitani-lab.ist.i.kyoto-u.ac.jp)