Perturb, Predict & Paraphrase: Semi-supervised Learning using Noisy Student for Image Captioning

This repository includes the implementation of "Perturb, Predict & Paraphrase: Semi-supervised Learning using Noisy Student for Image Captioning" to appear in the Proceedings of IJCAI (30th International Joint Conference on Artificial Intelligence), 2021.

Requirements

• Python 3.6
• Java 1.8.0
• PyTorch 1.0
• tensorboardX

Training

Prepare data

Download the datafiles and follow instructions given here

Start training

We have included a file pseudo_labels.txt which has the pseudo-labels generated by a teacher trained on 1% MSCOCO labeled data. This file can be used to train the student

$ CUDA_VISIBLE_DEVICES=0 sh train_student.sh

See opts.py for the options.

(Optional) Generate your own Pseudo Labels

To generate pseudo-labels, first train a teacher model using

$ CUDA_VISIBLE_DEVICES=0 sh train_teacher.sh

Then, generate pseudo-labels using

$ CUDA_VISIBLE_DEVICES=0 sh get_pseudo_labels.sh

Paraphraser

Refer to our paper for how to create a dataset for the paraphraser model. The code to train, and perform inference on the BART paraphraser can be found here.

1M unlabeled images

Follow https://github.com/ruotianluo/GoogleConceptualCaptioning for dataset creation and handling.

References

1. https://github.com/ruotianluo/self-critical.pytorch
2. https://github.com/husthuaan/AoANet
3. https://github.com/ruotianluo/GoogleConceptualCaptioning

Citation

If you use this code for your research, please consider citing our work

@inproceedings{ijcai2021-105,
  title     = {Perturb, Predict & Paraphrase: Semi-Supervised Learning using Noisy Student for Image Captioning},
  author    = {Jain, Arjit and Reddy Samala, Pranay and Jyothi, Preethi and Mittal, Deepak and Singh, Maneesh},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Zhi-Hua Zhou},
  pages     = {758--764},
  year      = {2021},
  month     = {8},
  note      = {Main Track}
  doi       = {10.24963/ijcai.2021/105},
  url       = {https://doi.org/10.24963/ijcai.2021/105},
}