Test-time Training for Matching-based Video Object Segmentation

[arXiv ] [project page]

This repository contains official code for our NeurIPS 2023 paper Test-Time Training for Matching-Based Video Object Segmentation.

What do we have here?

1. Installation

2. Data preparation

3. Test-time training

4. Citation

Installation

You can find below the installation script:

python -m venv ENV
source ENV/bin/activate
pip install torch torchvision
pip install pyyaml

Data preparation

We evaluated our test-time training strategy on four datasets:

• DAVIS 2017
• YOUTUBE-VOS 2018
• MOSE
• DAVIS-C

For more details on the datasets, please refer to DATA_PREPARATION.

Test-time training

We evaluated our proposed test-time training strategy starting from two offline-trained matching-based models:

• STCN
• XMem

Test-time training with the STCN model

For more details for please refer to STCN.

Test-time training with the XMem model

For more details for please refer to XMem.

Citation

If you use this code for your research, please consider citing our papers:

@inproceedings{bertrand2023ttt_vos,
  title={Test-time Training for Matching-based Video Object Segmentation},
  author={Bertrand, Juliette and Kordopatis-Zilos, Giorgos and Kalantidis, Yannis and Tolias, Giorgos},
  booktitle={Neural Information Processing Systems (NeurIPS)},
  year={2023}
}

Acknowledgement

We want to thank @hkchengrex for providing publicly available code and pretrained models for STCN and XMem.