🔥 Official implementation of "Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive" (ICLR 2024)
Our environment is built on top of ControlNet:
conda env create -f environment.yaml
conda activate aldm
pip install mit-semseg # for segmentation network UperNet
Pretrained models ade20k_step9.ckpt and cityscapes_step9.ckpt can be downloaded from here. They should be stored in the checkpoint folder.
Datasets should be structured as follows to enable ALDM training. Dataset path should be adjusted accordingly in dataloader/cityscapes.py, dataloader/ade20k.py and dataloader/coco_stuff.py. Check convert_coco_stuff_id.ipynb for converting coco stuff labels.
Click to expand
datasets
├── cityscapes
│ ├── gtFine
│ ├── train
│ └── val
│ └── leftImg8bit
│ ├── train
│ └── val
├── ADE20K
│ ├── annotations
│ ├── train
│ └── val
│ └── images
│ ├── train
│ └── val
├── COCOStuff
│ ├── train_img
│ ├── val_img
│ ├── train_label
│ ├── val_label
│ ├── train_label_convert # New: after converting
│ └── val_label_convert # New: after converting
└── ...
We provide three ways for testing: (1) JupyterNotebook, (2) Gradio Demo, (3) Bash scripts.
-
JupyterNotebook: we provided one sample layout for quick test without requiring dataset setup.
-
Run the command after the dataset preparation.
gradio gradio_demo/gradio_seg2image_cityscapes.py
- Bash scripts: we provide some bash scripts to enable large scale generation for the whole dataset. The synthesized data can be further used for training downstream models, e.g., semantic segmentation networks.
If you find our work useful, please star this repo and cite:
@inproceedings{li2024aldm,
title={Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive},
author={Li, Yumeng and Keuper, Margret and Zhang, Dan and Khoreva, Anna},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024}
}
This project is open-sourced under the AGPL-3.0 license. See the LICENSE file for details.
For a list of other open source components included in this project, see the file 3rd-party-licenses.txt.
This software is a research prototype, solely developed for and published as part of the publication cited above.
Please feel free to open an issue or contact personally if you have questions, need help, or need explanations. Don't hesitate to write an email to the following email address: liyumeng07@outlook.com