Skip to content
/ MBW Public

This work generates 2D and 3D landmark labels from videos with only two or three uncalibrated, handheld cameras moving in the wild. NeurIPS 2022.

License

Notifications You must be signed in to change notification settings

mosamdabhi/MBW

MBW: Multiview Bootstrapping in the Wild (NeurIPS 2022)

Multiview Bootstrapping in the Wild (MBW) provides a powerful way of generating labeled data in the wild at scale. Thereby, it democratizes the domain of data collection to a plethora of machine learning-driven computer vision applications via its novel self-supervision technique.

 

Requirements (if using GPU)

  • Tested in Pytorch 1.11, with CUDA 11.3

 

Setup

  1. Create a conda environment and activate it.
    conda env create -f environment_<cpu/gpu>.yml (change the flag within <> based on the available system)
    conda activate mbw
    pip install opencv-python
    
  2. Please do a clean install of the submodule robust_loss_pytorch:
    cd modules/helpers/robust_loss_pytorch
    pip install git+https://github.com/jonbarron/robust_loss_pytorch
    
  3. Please do a clean install of the submodule torch_batch_svd: (if using GPU)
    cd modules/helpers/torch-batch-svd
    export CUDA_HOME=/your/cuda/home/directory/    
    python setup.py install
    

 

Data & Pre-trained models

  1. Fetch the pre-trained flow and detector models from Zenodo using:

    zenodo_get 10.5281/zenodo.7054596
    unzip models.zip
    rm -rf models.zip && rm -rf md5sums.txt
    
  2. Download the data from DOI and unzip it in the data directory.

    zenodo_get 10.5281/zenodo.7058567
    unzip data.zip
    rm -rf data.zip && rm -rf md5sums.txt
    

    The final directory after retrieving pre-trained models and sample data should look like this:

    ${mbw}
     `-- data
         `-- Chimpanzee
             |-- annot/
             |-- images/
         
     `-- models
         |-- detector/
         |-- flow/
         |-- mvnrsfm/
    

 

Run unit tests

./scripts/unit_tests.sh

Training (Generate labels from MBW)

./scripts/train.sh

Evaluation and visualization

./scripts/eval.sh
./scripts/visualize.sh    

Demo (Jupyter notebook examples)

Please run demo.ipynb to play around and see visualizations of labels in the provided Jupyter notebook. We provide tools to visualize the 2D predictions (along with a confidence flag of Accept or Reject). The reconstructed 3D could also be visualized via an interactive plotly visualization tool.

Citation

If you use our code, dataset, or models in your research, please cite with:


@inproceedings{dabhi2022mbw,
	title={MBW: Multi-view Bootstrapping in the Wild},
	author={Dabhi, Mosam and Wang, Chaoyang and Clifford, Tim and Jeni, Laszlo and Fasel, Ian and Lucey, Simon},
	booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
	year={2022},
	ee = {https://openreview.net/forum?id=i1bFPSw42W0},
	organization={NeurIPS}
}


@inproceedings{dabhi2021mvnrsfm,
	title={High Fidelity 3D Reconstructions with Limited Physical Views},
	author={Dabhi, Mosam and Wang, Chaoyang and Saluja, Kunal and Jeni, Laszlo and Fasel, Ian and Lucey, Simon},
	booktitle={2021 International Conference on 3D Vision (3DV)},
	year={2021},
	ee = {https://ieeexplore.ieee.org/abstract/document/9665845},
	organization={IEEE}
}

About

This work generates 2D and 3D landmark labels from videos with only two or three uncalibrated, handheld cameras moving in the wild. NeurIPS 2022.

Topics

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published