Self-Adaptive Training

This is the PyTorch implementation of the

• NeurIPS'2020 paper Self-Adaptive Training: beyond Empirical Risk Minimization，
• Journal version Self-Adaptive Training: Bridging the Supervised and Self-Supervised Learning.

Self-adaptive training significantly improves the generalization of deep networks under noise and enhances the self-supervised representation learning. It also advances the state-of-the-art on learning with noisy label, adversarial training and the linear evaluation on the learned representation.

News

• 2021.10: The code of Selective Classification for SAT has been released here.
• 2021.01: We have released the journal version of Self-Adaptive Training, which is a unified algorithm for both the supervised and self-supervised learning. Code for self-supervised learning will be available soon.
• 2020.09: Our work has been accepted at NeurIPS'2020.

Requirements

• Python >= 3.6
• PyTorch >= 1.0
• CUDA
• Numpy

Usage

Standard training

The main.py contains training and evaluation functions in standard training setting.

Runnable scripts

• Training and evaluation using the default parameters

  We provide our training scripts in directory scripts/. For a concrete example, we can use the command as below to train the default model (i.e., ResNet-34) on CIFAR10 dataset with uniform label noise injected (e.g., 40%):

  $ bash scripts/cifar10/run_sat.sh [TRIAL_NAME]
  

  The argument TRIAL_NAME is optional, it helps us to identify different trials of the same experiments without modifying the training script. The evaluation is automatically performed when training is finished.

• Additional arguments

  • noise-rate: the percentage of data that being corrupted
  • noise-type: type of random corruptions (i.e., corrupted_label, Gaussian,random_pixel, shuffled_pixel)
  • sat-es: initial epochs of our approach
  • sat-alpha: the momentum term $\alpha$ of our approach
  • arch: the architecture of backbone model, e.g., resnet34/wrn34

Results on CIFAR datasets under uniform label noise

• Test Accuracy(%) on CIFAR10

Noise Rate	0.2	0.4	0.6	0.8
ResNet-34	94.14	92.64	89.23	78.58
WRN-28-10	94.84	93.23	89.42	80.13

• Test Accuracy(%) on CIFA100

Noise Rate	0.2	0.4	0.6	0.8
ResNet-34	75.77	71.38	62.69	38.72
WRN-28-10	77.71	72.60	64.87	44.17

Runnable scripts for repreducing double-descent phenomenon

You can use the command as below to train the default model (i.e., ResNet-18) on CIFAR10 dataset with 16.67% uniform label noise injected (i.e., 15% label error rate):

$ bash scripts/cifar10/run_sat_dd_parallel.sh [TRIAL_NAME]
$ bash scripts/cifar10/run_ce_dd_parallel.sh [TRIAL_NAME]

Double-descent ERM vs. single-descent self-adaptive training

Double-descent ERM vs. single-descent self-adaptive training on the error-capacity curve. The vertical dashed line represents the interpolation threshold.

Double-descent ERM vs. single-descent self-adaptive training on the epoch-capacity curve. The dashed vertical line represents the initial epoch E_s of our approach.

Adversarial training

We use state-of-the-art adversarial training algorithm TRADES as our baseline. The main_adv.py contains training and evaluation functions in adversarial training setting on CIFAR10 dataset.

Training scripts

• Training and evaluation using the default parameters

  We provides our training scripts in directory scripts/cifar10. For a concrete example, we can use the command as below to train the default model (i.e., WRN34-10) on CIFAR10 dataset with PGD-10 attack ($\epsilon$=0.031) to generate adversarial examples:

  $ bash scripts/cifar10/run_trades_sat.sh [TRIAL_NAME]
  

• Additional arguments

  • beta: hyper-parameter $1/\lambda$ in TRADES that controls the trade-off between natural accuracy and adversarial robustness
  • sat-es: initial epochs of our approach
  • sat-alpha: the momentum term $\alpha$ of our approach

Robust evaluation script

Evaluate robust WRN-34-10 models on CIFAR10 under PGD-20 attack:

  $ python pgd_attack.py --model-dir "/path/to/checkpoints"

This command evaluates 71-st to 100-th checkpoints in the specified path.

Results

Self-Adaptive Training mitigates the overfitting issue and consistently improves TRADES.

Attack TRADES+SAT

We provide the checkpoint of our best performed model in Google Drive and compare its natural and robust accuracy with TRADES as below.

Attack (submitted by) \ Method	TRADES	TRADES + SAT
None (initial entry)	84.92	83.48
PGD-20 (initial entry)	56.68	58.03
MultiTargeted-2000 (initial entry)	53.24	53.46
Auto-Attack+ (Francesco Croce)	53.08	53.29

Reference

For technical details, please check the conference version or the journal version of our paper.

@inproceedings{huang2020self,
  title={Self-Adaptive Training: beyond Empirical Risk Minimization},
  author={Huang, Lang and Zhang, Chao and Zhang, Hongyang},
  booktitle={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

@article{huang2021self,
  title={Self-Adaptive Training: Bridging the Supervised and Self-Supervised Learning},
  author={Huang, Lang and Zhang, Chao and Zhang, Hongyang},
  journal={arXiv preprint arXiv:2101.08732},
  year={2021}
}

Contact

If you have any question about this code, feel free to open an issue or contact laynehuang@pku.edu.cn.