CPGAN

Pytorch implementation for reproducing CPGAN results in the paper CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-image Synthesis by Jiadong Liang, Wenjie Pei, Feng Lu.

Getting Started

• Create anaconda virtual environment

conda create -n CPGAN python=2.7
conda activate CPGAN

• Install PyTorch and dependencies from http://pytorch.org

conda install pytorch=1.0.1 torchvision==0.2.1 cudatoolkit=10.0.130

• PIP Install

pip install python-dateutil easydict pandas torchfile nltk scikit-image h5py pyyaml

• Clone this repo:

https://github.com/dongdongdong666/CPGAN.git

• Download train2014-text.zip from here and unzip it to data/coco/text/.

• Unzip val2014-text.zip in data/coco/text/.

• Download memory features for each word from here and put the features in memory/.

• Download Inception Enocder , Generator , Text Encoder and put these models in models/.

Test

Sampling

• Set B_VALIDATION：False in "/code/cfg/eval_coco.yml".
• Run python eval.py --cfg cfg/eval_coco.yml --gpu 0 to generate examples from captions listed in "data/coco/example_captions.txt". Results are saved to "outputs/Inference_Images/example_captions".

Validation

• Set B_VALIDATION：True in "/code/cfg/eval_coco.yml".
• Run python eval.py --cfg cfg/eval_coco.yml --gpu 0 to generate examples for all captions in the validation dataset. Results are saved to "outputs/Inference_Images/single".
• Compute inception score for the model trained on coco.

python caculate_IS.py --dir ../outputs/Inference_Images/single/

• Compute R precison for the model trained on coco.

python caculate_R_precison.py --cfg cfg/coco.yml --gpu 0 --valid_dir ../outputs/Inference_Images/single/

Train

TBA

Examples generated by CPGAN

• Qualitative comparison between different modules of our model for ablation study, the results of AttnGAN are also provided for reference.

• Qualitative comparison between our CPGAN with other classical models for text-to-image synthesis.

Performance

Note that after cleaning and refactoring the code of the paper, the results are slightly different. We use the Pytorch implementation to measure inception score and R precision.

Model	R-precision↑	IS↑
Reed	(-)	7.88 ± 0.07
StackGAN	(-)	8.45 ± 0.03
Infer	(-)	11.46 ± 0.09
SD-GAN	(-)	35.69 ± 0.50
MirrorGAN	(-)	26.47 ± 0.41
SEGAN	(-)	27.86 ± 0.31
DMGAN	88.56%	30.49 ± 0.57
AttnGAN	82.98%	25.89 ± 0.47
objGAN	91.05%	30.29 ± 0.33
CPGAN	93.59%	52.73 ± 0.61

Citing CPGAN

If you find CPGAN useful in your research, please consider citing:

@misc{liang2019cpgan,
    title={CPGAN: Full-Spectrum Content-Parsing Generative Adversarial Networks for Text-to-Image Synthesis},
    author={Jiadong Liang and Wenjie Pei and Feng Lu},
    year={2019},
    eprint={1912.08562},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}