Geometry-biased Transformers for Novel View Synthesis

Paper    Web    Demo

For detailed instructions refer to SETUP.md

Follow instructions from the official CO3D repository to download the dataset in this format.

Train GBT model on 10 categories (category agnostic)

python scripts/train.py --config-path configs/cat_agnostic_gbt.yaml

Train GBT-nb (no geometric bias) model on 10 categories (category agnostic)

python scripts/train.py --config-path configs/cat_agnostic_gbt_nb.yaml

Note: Modify yaml config files with appropriate num_pixel_queries that can fit on the GPU.

Download pre-trained checkpoints from this link. Extract contents inside the repository base directory. Alternatively, run the following commands from terminal.

pip install gdown
gdown 1eHeNba_qlsM-7iEiIlZw9XH9-VXqem7T
unzip runs.zip
rm runs.zip

Verify that the extracted checkpoints are of the following structure.

gbt/runs/co3dv2/cat_agnostic/
|-- gbt
|   `-- latest.pt
`-- gbt_nb
    `-- latest.pt

Run GBT model trained on 10 categories (category agnostic)

python scripts/infer.py --config-path configs/cat_agnostic_gbt.yaml --dataset-path /path/to/co3d/dataset --category donut

Run GBT-nb (no geometric bias) model trained on 10 categories (category agnostic)

python scripts/infer.py --config-path configs/cat_agnostic_gbt_nb.yaml --dataset-path /path/to/co3d/dataset --category donut

The inference script computes average psnr and lpips metrics for objects of the specified category, and also saves individual rotating gifs for qualitative analysis.

runs/co3dv2/cat_agnostic/gbt/infer/num_views=3/donut/
|-- 198_21296_42378.gif
|-- 290_30761_58510.gif
|-- ...
`-- metrics.txt