Semantic Part Detection via Matching: Learning to Generalize to Novel Viewpoints from Limited Training Data

This repo constains the pytorch implementation for the ICCV2019: Semantic Part Detection via Matching: Learning to Generalize to Novel Viewpoints from Limited Training Data arxiv.

@inproceedings{bai2019semantic,
  title={Semantic Part Detection via Matching: Learning to Generalize to Novel Viewpoints from Limited Training Data},
  author={Bai, Yutong and Liu, Qing and Xie, Lingxi and Qiu, Weichao and Zheng, Yan and Yuille, Alan L},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  pages={7535--7545},
  year={2019}
}

Highlight

• We present an approach which can learn from a small annotated dataset containing a limited range of viewpoints and generalize to detect semantic parts for a much larger range of viewpoints.
• The approach is based on our matching algorithm, which is used for finding accurate spatial correspondence between two images and transplanting semantic parts annotated on one image to the other.

Usage

Our code uses PyTorch to implement. Experiments are performed on the VehicleSemanticPart (VSP) dataset and synthetic images generated by UnrealCV. Please refer to these websites for details of data preparation.

• supports python3 and pytorch=1.1.0

• need to install MATLAB

• clone this repo: git clone https://github.com/ytongbai/SemanticPartDetection.git

• Pipeline: For both training and testing stage:

  • Generate synthetic data.

  • Get training images from different viewpoints.

    python get_vp_examples.py, python assign_synthetic_img.py

  • Extract features from ImageNet pretrained VGG-16 Network.

    python extractLayerFeat.py

  • Data format change to MATLAB.

    python featPickle2featCell.py

  • Use maximal clique algorithm to conduct the matching stage.

    bash MaximalCliqueAlgorithm/run_per_folder.sh

  • Use feature extracted from pool-3 layer from ImageNet Pretrained VGG-16 to refine.

    python refine_pool3.py

  • Aggregate 2D parts to 3D CAD model.

    python aggregate_3d_wPars.py,python solve_nearest_score.py

  • Calculate mAP score.

    python calculate_mAP.py

Contact

For any questions, please feel free to reach

Yutong Bai: ytongbai@gmail.com