This is the implementation of our papers below, including the code and dataset (DADA-seg).
ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Data, IROS 2021, [paper].
Exploring Event-Driven Dynamic Context for Accident Scene Segmentation, T-ITS 2021, [paper].
- [04/12/2022] Generated event data release.
- [12/27/2021] Initial release.
The requirements are listed in the requirement.txt file. To create your own environment, an example is:
conda create -n issafe python=3.7
conda activate issafe
cd /path/to/ISSAFE
pip install -r requirement.txtFor the basic setting of this work, please prepare datasets of Cityscapes, and DADA-seg.
Our proposed DADA-seg dataset is a subset from DADA-2000.
The DADA-seg dataset and event data of Cityscapes are now available:
| Dataset | Image & label | Event | Label-only |
|---|---|---|---|
| DADA-seg (val) | 46.8MB [Baidu Drive] | 401.2MB [Baidu Drive] | 2.7MB [Google Drive] |
| Cityscapes (train + val) | - | 7.5GB [Baidu Drive] | - |
Note: The event data is generated by EventGAN. The anchor and its previous frames are needed.
A structure of dataset should be:
dataset
├── Cityscapes
│ ├── event
│ │ ├── train
│ │ │ ├─aachen
│ │ │ │ ├─aachen_000000_000019_gtFine_event.npz # event volume
│ │ └── val
│ ├── gtFine
│ │ ├── train
│ │ └── val
│ ├─leftImg8bit_prev # for event synthesic
│ │ ├─train
│ │ │ ├─aachen
│ │ │ │ ├─aachen_000000_000019_leftImg8bit_prev.png
│ │ └─val
│ ├── leftImg8bit
│ │ ├── train
│ └── └── val
└── DADA_seg
├── dof
│ └── val
├── event
│ └── val
├── gtFine
│ └── val
└── leftImg8bit
├── train
└── val
(optional) other dataset sources used in EDCNet: BDD3K, KITTI-360, ApolloScape.
(optional) other modalities: dense optical flow.
The model of EDCNet can be found at models/edcnet.py.
Before run the training script, please modify your own path configurations at mypath.py.
The training configurations can be adjusted at train.py.
An example of training is python train.py
The evaluation configurations can be adjusted at eval.py.
To achieve the evaluation result of EDCNet in D2S mode with 2 event time bins, the weights can be downloaded in Google Drive.
Put the weight at run/cityscapesevent/test_EDCNet_r18/model_best.pth.
An example of evaluation of the EDCNet at B=2 event time bins is python eval.py.
This repository is under the Apache-2.0 license. For commercial use, please contact with the authors.
If you are interested in this work, please cite the following work:
@INPROCEEDINGS{zhang2021issafe,
author={Zhang, Jiaming and Yang, Kailun and Stiefelhagen, Rainer},
booktitle={2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
title={ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Data},
year={2021},
pages={1132-1139},
doi={10.1109/IROS51168.2021.9636109}}
@ARTICLE{zhang2021edcnet,
author={Zhang, Jiaming and Yang, Kailun and Stiefelhagen, Rainer},
journal={IEEE Transactions on Intelligent Transportation Systems},
title={Exploring Event-Driven Dynamic Context for Accident Scene Segmentation},
year={2021},
pages={1-17},
doi={10.1109/TITS.2021.3134828}}