Squeeze-and-Span

Yu Yang*, Xiaotian Cheng*, Chang Liu, Hakan Bilen, Xiangyang Ji. Distilling Representations from GAN Generator via Squeeze and Span. In NeurIPS 2022. [pdf, bibtex]

Dataset

Put data or create a soft link to the dataset root directory in ./data/. For example,

data/
├── CIFAR10
│   ├── cifar-10-batches-py
│   └── cifar-10-python.tar.gz
├── CIFAR100
│   ├── cifar-100-python
│   └── cifar-100-python.tar.gz
├── STL10
│   ├── stl10_binary
│   └── stl10_binary.tar.gz

Please download dataset from CIFAR, STL10.

Pre-trained GANs

Please download pre-trained GAN checkpoints from

File	Url
cifar10u-cifar-ada-best-fid.pkl
cifar100u-cifar-best-fid4.13.pkl
stl10u-my128-best-fid20.86.pkl

checkpoints/
├── cifar100u-cifar-best-fid4.13.pkl
├── cifar10u-cifar-ada-best-fid.pkl
└── stl10u-my128-best-fid20.86.pkl

Experiment scripts

Table 1: Representation transfer from different teachers

Knowledge Source	Transfer Method	Domain	CIFAR10	CIFAR100
Discriminator	Direct use (single feature)	Syn. & Real	63.81 [script]	30.11 [script]
Discriminator	Direct use (multi-feature)	Syn. & Real	77.58 [script]	51.63 [script]
Latent variable	Encoding	Syn.	57.15 [script]	32.19 [script]
Latent variable	Encoding	Syn. & Real	50.27 [script]	28.43 [script]
Latent variable	Vanilla distillation (w/ aug)	Syn.	84.84 [script]	53.26 [script]
Latent variable	Squeeze	Syn.	86.99 [script]	58.56 [script]
Latent variable	Squeeze and span	Syn. & Real	90.95	66.17
Generator feature	Vanilla distillation (w/ aug)	Syn.	84.48 [script]	52.77 script]
Generator feature	Squeeze	Syn.	87.67 [script]	57.35 [script]
Generator feature	Squeeze and span	Syn. & Real	92.54 [script]	67.87 [script]

Table 2: Comparison to SSL

Pretrain Data	Methods	CIFAR10	CIFAR100	STL10
Real	SimSiam	90.94 [script]	62.44 [script]	71.30 [script]
Real	VICReg	89.20 [script]	63.31 [script]	74.43 [script]
Syn	SimSiam	85.11 [script]	47.89 [script]	73.38 [script]
Syn	VICReg	84.68 [script]	52.84 [script]	70.80 [script]
Syn	Squeeze (Ours)	87.67 [script]	57.35 [script]	73.35 [script]
Real & Syn	SimSiam	90.88 [script]	62.68 [script]	71.70 [script]
Real & Syn	VICReg	90.46 [script]	65.22 [script]	75.05 [script]
Real & Syn	Squeeze & Span (Ours)	92.54 [script]	67.87 [script]	76.83 [script]

Table 5: Ablation study

	$\mathcal{L}_{\text{RD}}$	$\mathcal{A}$	$T_\phi$	$\mathcal{L}_{\text{var}}$	$\mathcal{L}_{\text{cov}}$	Span	Top-1 Acc
a	$\checkmark$						74.20 [script]
b	$\checkmark$	$\checkmark$					84.48 [script]
c	$\checkmark$	$\checkmark$	$\checkmark$				10.00 [script]
d	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$			79.10 [script]
e	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$		87.67 [script]
f	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	92.54 [script]

Visualization as in Fig 2

CUDA_VISIBLE_DEVICES=0 python plot_umap --output-dir=output/plot_umap --gpath=checkpoints/cifar10u-cifar-ada-best-fid.pkl --cpath=checkpoints/cifar10_wrn.pth

Plot Fig 4

python paper_plots/acc_vs_fid.py

Evaluatiion

Dataset	Script
CIFAR10	scripts/cifar10_eval_linear.sh
CIFAR100	scripts/cifar100_eval_linear.sh
STL10	scripts/stl10_eval_linear.sh

Citation

If you find this repository useful in your research, please consider citing:

@inproceedings{
    yang2022distilling,
    title={Distilling Representations from {GAN} Generator via Squeeze and Span},
    author={Yu Yang* and Xiaotian Cheng* and Chang Liu and Hakan Bilen and Xiangyang Ji},
    booktitle={Advances in Neural Information Processing Systems},
    editor={Alice H. Oh and Alekh Agarwal and Danielle Belgrave and Kyunghyun Cho},
    year={2022},
    url={https://openreview.net/forum?id=_P4JCoz83Mb}
}