Chao-Yuan Wu, Justin Johnson, Jitendra Malik, Christoph Feichntenhofer, Georgia Gkioxari
mcc.mp4
This is a PyTorch implementation of Multiview Compressive Coding (MCC):
@article{wu2023multiview,
author = {Wu, Chao-Yuan and Johnson, Justin and Malik, Jitendra and Feichtenhofer, Christoph and Gkioxari, Georgia},
title = {Multiview Compressive Coding for 3{D} Reconstruction},
journal = {arXiv preprint arXiv:2301.08247},
year = {2023},
}
This repo is a modified from the MAE repo. Installation and preparation follow that repo. Please also install PyTorch3D for 3D related funcionality.
pip install h5py omegaconf submitit
Please see DATASET.md for information on data preparation.
To train an MCC model on the CO3D v2 dataset, please run
python submitit_mcc.py \
--use_volta32 \
--job_dir ./output \
--nodes 4 \
--co3d_path [path to CO3D dataset] \
--resume [path to pretrained weights for RGB encoder (optional)] \
--holdout_categories \
- With
--holdout_categories, we hold out a subset of categories during training, and evaluate on the held out categories only. To train on all categories, please remove the argument. - Here we use 4 nodes (machines) by default; Users may use a different value.
- Optional: The RGB encoder may be initialized by a pre-trained image model. An ImageNet1K-MAE-pretrained model is available [here]. Using a pre-trained model may speed up training but does not affect the final results much.
To train an MCC model on the Hypersim dataset, please run
python submitit_mcc.py \
--use_volta32 \
--job_dir ./output \
--nodes 4 \
--hypersim_path [path to Hypersim dataset] \
--resume [path to pretrained weights for RGB encoder (optional)] \
--use_hypersim \
--viz_granularity 0.2 --eval_granularity 0.2 \
--blr 5e-5 \
--epochs 50 \
--train_epoch_len_multiplier 3200 \
--eval_epoch_len_multiplier 200 \
- Here we additionally specify
--use_hypersimfor running Hypersim scene reconstruction experiments. - We use slightly different hyperparameters to accommodate the scene reconstruction task.
To test on iPhone captures, please use the Record3D App on an iPhone to capture an RGB image and the corresonding point cloud (.obj) file. To generate the segmentation mask, we used a private segmentation model; Users may use other tools/models to obtain the mask. Two example captures are available in the demo folder.
To run MCC inference on the example, please use, e.g.,
python demo.py --image demo/quest2.jpg --point_cloud demo/quest2.obj --seg demo/quest2_seg.png \
--checkpoint [path to model checkpoint] \
One may use a checkpoint from the training step above or download a model that is already trained on all CO3D v2 categories [here].
One may set the --score_thresholds argument to specify the score thresholds (More points are shown with a lower threshold, but the predictions might be noisier).
The script will generate an html file showing an interactive visualizaion of the MCC output with plotly.
Part of this implementation is based on the MAE codebase. We thank Sasha Sax for help on loading Hypersim and Taskonomy data.
Multiview Compressive Coding is released under the CC-BY-NC 4.0.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent