The intuition of this paper is simple: why don't we assign Mixup samples to a generalized negative class? Just as humans may perceive, if an animal is half dog and half cat, it is actually of neither species.
With this intuition, our framework is like this:Here we present the implementation of UniCon(ours)[1], SupCon[2], SimClr[3], Xent and SupMix(a hybrid of SupCon and
Un-Mix[4] proposed by us) on Pytorch. We use CIFAR-100 as an example dataset.
[1]: Universum-inspired Supervised Contrastive Learning paper
[2]: Supervised Contrastive Learning paper
[3]: A Simple Framework for Contrastive Learning of Visual Representations paper
[4]: Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation
Learning paper
Results on CIFAR-100:
Method | Architecture | Batch size | Accuracy(%) |
---|---|---|---|
Cross Entropy | ResNet-50 | 1024 | 74.6 |
SimCLR | ResNet-50 | 1024 | 68.4 |
SupCon(baseline) | ResNet-50 | 1024 | 76.5 |
UniCon(ours) | ResNet-18 | 256 | 79.2 |
UniCon(ours) | ResNet-50 | 256 | 81.7 |
We refer you to the official pytorch implementation of SupCon for
the implementations of SupCon and SimCLR. If you want to run SupCon and SimCLR with our code, you can simply change
python main_supcon.py
to python main_unicon.py
and maintain other parts. We will give instructions on SupCon,
SupMix and CE classifier. Our samples are running on CIFAR-100 and you can run on CIFAR-10, TinyImageNet and ImageNet
(you need to download the datasets yourself for the latter two) with --dataset cifar10/tinyimagenet/imagenet
.
(1) UniCon
python main_unicon.py --batch_size 256
--learning_rate 0.05 --temp 0.1
--cosine --warm
You can change Mixup parameter with --lamda 0.5
. Or you can use CutMix with --mix cutmix
.
(2) SupMix
python main_supmix.py --batch_size 256
--learning_rate 0.05 --temp 0.1
--cosine --warm --beta --lamda 0.5
Here --lamda
is used in distribution Beta(lambda, lambda). You can also set lambda to a constant by reducing
--beta
.
(3) Cross Entropy
python main_ce.py --batch_size 1024
--learning_rate 0.8
--cosine
You can use --augment
, --mixup --alpha 0.5
, --cutmix --alpha 0.5 --cutmix_prob 0.5
to enable augmentations,
Mixup and CutMix, respectively. Please note that Mixup and CutMix cannot be applied together. If so, only Mixup is
used.
(4) Linear Evaluation
python main_linear.py --batch_size 512
--learning_rate 5
--ckpt /path/to/model.pth
@article{han2022universum,
title={Universum-inspired Supervised Contrastive Learning},
author={Han, Aiyang and Chen, Songcan},
journal={arXiv preprint arXiv:2204.10695},
year={2022}
}