GAIS-Net

GAIS-Net: Geometry-Aware Instance Segmentation with Disparity Maps

Cho-Ying Wu, Xiaoyan Hu, Michael Happold, Qiangeng Xu, Ulrich Neumann, CVPR Workshop on Scalability in Autonomous Driving, 2020.

Advantage:

👍 The first work to directly regress instances from depth maps to advance the multi-modal learning for outdoor scenarios.

👍 High performance and highly modulized. The codes are based on the mask-rcnn benchmark framework.

Please visit out Project site for paper and intorduction.

This project is developed upon Mask-RCNN and is done during an internship at Argo AI

Feature: Resolve overlapping areas between instances by introducing geometry information

Installation

Check INSTALL.md for installation instructions.

Note that cocoapi, cityscapesScripts, and apex are needed for the evaluation.

Implementation

The geometry-aware fusion module is implemented under roi_heads.py mask_head.py maskiou_head.py

Check these files and related files for the features we implement.

Data

1. The pre-generated stereo disparity maps by PSMNet for Cityscapes could be downloaded here. Please first create folders "datasets/cityscapes" at the root of the repo and extract the zip file under datasets/cityscapes/

2. Go to Cityscapes to download images of train/val set. Put the images under "datasets/cityscapes/train/image_2" and also "val". Download the annotations and put under datasets/cityscapes/annotations/

Evaluation

NUM_GPUS=4 python -m torch.distributed.launch --nproc_per_node=$NUM_GPUS ./tools/test_net.py --config-file "configs/cityscapes_v4.yaml" TEST.IMS_PER_BATCH 4

Pretrain model

The pretrained weights could be downloaded here put the .pth file under "ckpt/"

Results

Citations

If you find the work useful, please condisder to cite:

@inproceedings{wu2020Cvprw,
title={Geometry-Aware Instance Segmentation with Disparity Maps},
author={Wu, Cho-Ying and Hu, Xiaoyan and Happold, Michael and Xu, Qiangeng and Neumann, Ulrich},
booktitle={CVPR Workshop on Scability in Autonomous Driving},
year={2020}
}

Thanks to the Third Parties

Thank Mask-RCNN and Mask Scoring RCNN