Skip to content

Official PyTorch implementation of GBT: Geometry-biased Transformers for Novel View Synthesis

Notifications You must be signed in to change notification settings

mayankgrwl97/gbt

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Geometry-biased Transformers for Novel View Synthesis

Geometry-biased Transformers for Novel View Synthesis

Paper    Web    Demo

Open in Colab

Environment Setup

For detailed instructions refer to SETUP.md

Dataset

Follow instructions from the official CO3D repository to download the dataset in this format.

Training

Commands

Train GBT model on 10 categories (category agnostic)

python scripts/train.py --config-path configs/cat_agnostic_gbt.yaml

Train GBT-nb (no geometric bias) model on 10 categories (category agnostic)

python scripts/train.py --config-path configs/cat_agnostic_gbt_nb.yaml

Note: Modify yaml config files with appropriate num_pixel_queries that can fit on the GPU.

Inference

Checkpoints

Download pre-trained checkpoints from this link. Extract contents inside the repository base directory. Alternatively, run the following commands from terminal.

pip install gdown
gdown 1eHeNba_qlsM-7iEiIlZw9XH9-VXqem7T
unzip runs.zip
rm runs.zip

Verify that the extracted checkpoints are of the following structure.

gbt/runs/co3dv2/cat_agnostic/
|-- gbt
|   `-- latest.pt
`-- gbt_nb
    `-- latest.pt

Commands

Run GBT model trained on 10 categories (category agnostic)

python scripts/infer.py --config-path configs/cat_agnostic_gbt.yaml --dataset-path /path/to/co3d/dataset --category donut

Run GBT-nb (no geometric bias) model trained on 10 categories (category agnostic)

python scripts/infer.py --config-path configs/cat_agnostic_gbt_nb.yaml --dataset-path /path/to/co3d/dataset --category donut

Output

The inference script computes average psnr and lpips metrics for objects of the specified category, and also saves individual rotating gifs for qualitative analysis.

runs/co3dv2/cat_agnostic/gbt/infer/num_views=3/donut/
|-- 198_21296_42378.gif
|-- 290_30761_58510.gif
|-- ...
`-- metrics.txt

About

Official PyTorch implementation of GBT: Geometry-biased Transformers for Novel View Synthesis

Resources

Stars

Watchers

Forks