Recurrent Neural Networks for Semantic Instance Segmentation

See the paper in arXiv here.

Installation

• Clone the repo:

git clone https://github.com/imatge-upc/rsis.git

• Install requirements pip install -r requirements.txt
• Install PyTorch 0.2 (choose the whl file according to your setup):

pip install http://download.pytorch.org/whl/cu80/torch-0.2.0.post3-cp27-cp27mu-manylinux1_x86_64.whl  
pip install torchvision

• Compile COCO Python API and add it to your PYTHONPATH:

cd src/coco/PythonAPI;
make

# Run from the root directory of this project
export PYTHONPATH=$PYTHONPATH:./src/coco/PythonAPI

Data

Pascal VOC 2012

• Download Pascal VOC 2012:

wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
tar -xvf VOCtrainval_11-May-2012.tar

• Download additional VOC annotations from Hariharan et al. Semantic contours from inverse detectors. ICCV 2011:

# berkeley augmented Pascal VOC
wget http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz # 1.3 GB
tar zxvf benchmark.tgz

• Create a merged dataset out of the two sets of images and annotations:

python src/dataloader/pascalplus_gen.py --voc_dir /path/to/pascal --contours_dir /path/to/additional/dataset --vocplus_dir /path/to/merged

• Precompute instance and semantic segmentation masks & ground truth files in COCO format:

python src/dataloader/pascal_precompute.py --split train --pascal_dir /path/to/merged

You must run this three times for the different splits (train, val and test).

Point args.pascal_dir to /path/to/merged.

CVPPP

Download the training CVPPP dataset from their website. In our case we just worked with the A1 dataset. Extract the A1 package and point args.leaves_dir to this folder. To obtain the test set for evaluation you will have to contact the organizers.

Cityscapes

Download the Cityscapes dataset from their website. Extract the images and the labels into the same directory and point args.cityscapes_dir to it.

Training

• Train the model with python train.py -model_name model_name. Checkpoints and logs will be saved under ../models/model_name.
• Other arguments can be passed as well. For convenience, scripts to train with typical parameters are provided under scripts/.
• Visdom can be enabled to monitor training losses and outputs:
  • First run the visdom server withpython -m visdom.server.
  • Run training with the --visdom flag. Navigate to localhost:8097 to visualize training curves.
• Plot loss curves at any time with python plot_curves.py -model_name model_name.

Evaluation

We provide bash scripts to display results and evaluate models for the three datasets. You can find them under the scripts folder.

In the case of cityscapes, the evaluation bash script will generate the results in the appropiate format to use the official evaluation code.

For CVPPP, the evaluation bash script will generate the results in the appropiate format to use the evaluation scripts that are provided with the dataset.

Pretrained models

Download weights for models trained with:

• Pascal VOC 2012
• Cityscapes
• CVPPP

Extract and place the obtained folder under models directory. You can then run evaluation scripts with the downloaded model by setting args.model_name to the name of the folder.

Contact

For questions and suggestions use the issues section or send an e-mail to amaia.salvador@upc.edu

Additional notes to GPI users

Helpful commands to train on the GPI cluster and get visualizations in your browser:

• Start server: with srun --tunnel $UID:$UID python -m visdom.server -port $UID.
• Check the node where the server launched and (eg. c3).
• Run training with:

srun --gres=gpu:1,gmem:12G --mem=10G python train.py --visdom -port $UID -server http://c3

Notice that the port and the server must match the ones used in the previous run.

• echo $UID to know which port you are using.
• ssh tunnel (run this in local machine): ssh -L 8889:localhost:YOUR_UID -p2222 user@imatge.upc.edu.
• Navigate to localhost:8889 in your browser locally.