Sketch-GAN

CVPR | paperswithcode

Pytorch implementation of the SketchGAN paper.
SketchGAN: Joint Sketch Completion and Recognition with Generative Adversarial Network
Fang Liu^1,2, Xiaoming Deng¹, Yu-Kun Lai³, Yong-Jin Liu^{4 *}, Cuixia Ma^{1,2 *}, Hongan Wang^{1,2 *},
¹State Key Laboratory of Computer Science and Beijing Key Lab of Human-Computer Interaction, Institute of Software, Chinese Academy of Sciences, ²University of Chinese Academy of Sciences, ³Cardiff University, ⁴Tsinghua University
In CVPR 2019.

Overview

SketchGAN is a new generative adversarial network (GAN) based approach that jointly completes and recognizes a sketch, boosting the performance of both tasks. It is used for hand-drawn sketch recognition which is a fundamental problem in computer vision, widely used in sketch-based image and video retrieval, editing, and reorganization¹.

The paper proposes a cascade Encode-Decoder network to complete the input sketch in an iterative manner, and employs an auxiliary sketch recognition task to boost the performance of both tasks¹.

Prerequisites

• Python 3
• CPU or NVIDIA GPU + CUDA CuDNN

Getting Started

Installation

• Clone this repo:

  git clone git@github.com:Ashish-Abraham/Sketch_GAN.git
  cd Sketch_GAN

• Install PyTorch and other dependencies (e.g., torchvision).

  pip install -r requirements.txt

Data Preparation

• Training image directory should be of the following structure

├── image_paths.csv

├── trainA
  ....
├── trainB
  ....
├── valA
  ....
└── valB
  ....

• Folders with A suffix contain corrupted sketches and those with B contain target sketches.
• The GAN should map A->B.
• Each image.png in __B folder contains corresponding imagec.png in __A folder.
• Use data_csv_script.py to generate corresponding csv file from dataset for training.

Model Training

The repo contains all code required to implement the GAN. Edit the scripts given or export the code in scripts to a jupyter notebook appropriately to train. Make sure cuda is available.

Initial Results

Input	Ground Truth	Output

Citation

F. Liu, X. Deng, Y. -K. Lai, Y. -J. Liu, C. Ma and H. Wang, "SketchGAN: Joint Sketch Completion and Recognition With Generative Adversarial Network," 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 5823-5832, doi: 10.1109/CVPR.2019.00598.

Yu, Q., Yang, Y., Liu, F. et al. Sketch-a-Net: A Deep Neural Network that Beats Humans. Int J Comput Vis 122, 411–425 (2017). https://doi.org/10.1007/s11263-016-0932-3