Skip to content

[3DV 2024] Repository for "Multi-Body Neural Scene Flow", in International Conference on 3D Vision 2024.

License

Notifications You must be signed in to change notification settings

kavisha725/MBNSF

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MBNSF

This repository is the official open-source implementation of the paper MBNSF - 3DV'2024 (oral):

Multi-Body Neural Scene Flow
Kavisha Vidanapathirana, Shin-Fang Ch'ng, Xueqian Li, Simon Lucey
International Conference on 3D Vision 2024 arXiv

This repository contains the code for:

Method overview

MBNSF is a regularizor for incorporating multi-body rigidity into neural scene flow in a manner that (1) doesn't require explicit estimaiton of SE(3) parameters for each rigid body, (2) is robust to the inherent noise in real-world LiDAR datasets, (3) maintains the fully unsupervised nature, and (4) maintains the estimation of continuous flow fields.


Prerequisites

Environment dependencies

Set up base environment
  • Create conda environment with python:
conda create --name mbnsf python=3.9.4
conda activate mbnsf
  • Install PyTorch with suitable cudatoolkit version. See here:
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 -c pytorch
# Make sure the pytorch cuda version matches your output of 'nvcc --version'
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install -c bottler nvidiacub
conda install pytorch3d -c pytorch3d
pip install open3d
  • Test installation using:
python -c "import torch ; import pytorch3d ; import open3d ; print(torch.cuda.is_available())"

Datasets

This repository provides evaluation scripts for the Argoverse dataset.

Download Argoverse test set:

Our test set for Argoverse consists of 18 sequences with 25 consecutive frames for evaluating long-term trajectories (which also results in 450 pairs for scene flow evaluation)

  • Download the Argoverse test set from here (~1.5 GB).
  • (Optional) The code for preparing this test set is provided in utils/get_gt_traj_argoverse.py.
For the Waymo test set:

For Waymo, we use the same test set as provided in FNSF.

  • Download the Waymo test set from here.
  • (Optional) Instructions for preparing this test set are provided here.

Evaluation

This section re-creates the results of Tab. 1 and Tab. 2 in our paper on the Argoverse dataset.

3D Scene Flow Prediction

In this section we optimize a scene flow field for a given pair of point clouds.

cd scene_flow_estimation/
  • Scene flow optimization using NSFP (baseline):
python nsfp.py --dataset_path </path/to/data>
  • Scene flow optimization using MBNSF (ours):
python mbnsf.py --dataset_path </path/to/data>
4D Trajectory Prediction

In this section we optmize a trajectory field for a sequence of point clouds.

cd trajectory_estimation/
  • Long-Tem trajectory optimization using NSFP (baseline) + Forward Euler integration:
python nsfp_fe.py --exp_name nsfp_fe_test --dataset_path </path/to/data>
python compute_traj_metrics.py --exp_name nsfp_fe_test --dataset_path </path/to/data>
  • Long-Tem trajectory optimization using MBNSF (ours) + Forward Euler integration:
python mbnsf_fe.py --exp_name mbnsf_fe_test --dataset_path </path/to/data>
python compute_traj_metrics.py --exp_name mbnsf_fe_test --dataset_path </path/to/data>
  • Long-Tem trajectory optimization using NTP (baseline, our implementation):
python ntp.py --exp_name ntp_test --dataset_path </path/to/data>
python compute_traj_metrics.py --exp_name ntp_test --dataset_path </path/to/data>
  • Long-Tem trajectory optimization using MBNT (ours): NTP + our regularizor:
python mbnt.py --exp_name mbnt_test --dataset_path </path/to/data>
python compute_traj_metrics.py --exp_name mbnt_test --dataset_path </path/to/data>


Citation

If you find this work useful in your research, please cite:

@inproceedings{vidanapathirana2023mbnsf,
  title={Multi-Body Neural Scene Flow},
  author={Vidanapathirana, Kavisha and Chng, Shin-Fang and Li, Xueqian and Lucey, Simon},
  booktitle={2024 International Conference on 3D Vision (3DV)},
  year={2024},
  organization={IEEE}
}

Acknowledgement

Functions from 3rd party have been acknowledged at the respective function definitions or readme files.

This project was mainly inspired by NSFP and SpectralMatching.