shape-of-motion's Introduction

Shape of Motion: 4D Reconstruction from a Single Video

Qianqian Wang^1,2*, Vickie Ye¹*, Hang Gao¹*, Jake Austin¹, Zhengqi Li², Angjoo Kanazawa¹

¹UC Berkeley ²Google Research

* Equal Contribution

Installation

git clone --recurse-submodules https://github.com/vye16/shape-of-motion
cd shape-of-motion/
conda create -n som python=3.10
conda activate som

Update requirements.txt with correct CUDA version for PyTorch and cuUML, i.e., replacing cu122 and cu12 with your CUDA version.


pip install -r requirements.txt
pip install git+https://github.com/nerfstudio-project/gsplat.git

Usage

Preprocessing

We depend on the third-party libraries in preproc to generate depth maps, object masks, camera estimates, and 2D tracks. Please follow the guide in the preprocessing README.

Fitting to a Video

python run_training.py --work-dir <OUTPUT_DIR> --data:davis --data.seq-name horsejump-low

Evaluation on iPhone Dataset

First, download our processed iPhone dataset from this link. To train on a sequence, e.g., paper-windmill, run:

python run_training.py --work-dir <OUTPUT_DIR> --port <PORT> --data:iphone --data.data-dir </path/to/paper-windmill/>

After optimization, the numerical result can be evaluated via:

PYTHONPATH='.' python scripts/evaluate_iphone.py --data_dir </path/to/paper-windmill/> --result_dir <OUTPUT_DIR> --seq_names paper-windmill

Citation

@inproceedings{som2024,
  title     = {Shape of Motion: 4D Reconstruction from a Single Video},
  author    = {Wang, Qianqian and Ye, Vickie and Gao, Hang and Austin, Jake and Li, Zhengqi and Kanazawa, Angjoo},
  journal   = {arXiv preprint arXiv:2407.13764},
  year      = {2024}
}

Recommend Projects