anything-of-anything / anything-3d Goto Github PK

View Code? Open in Web Editor NEW

1.5K 35.0 74.0 55.29 MB

Segment-Anything + 3D. Let's lift anything to 3D.

License: MIT License

Python 99.96% Shell 0.04%

3d computer-vision reconstruction segment segment-anything

anything-3d's Introduction

🎉🎉🎉Welcome to the Anything-3D GitHub repository!🎉🎉🎉

🔥Technical Report is out🔥
🔥Open-source Code is out🔥

Here we present a project where we combine Segment Anything with a series of 3D models to create a very interesting demo. This is currently a small project, but we plan to continue improving it and creating more exciting demos.

Contributions are highly Welcomed!🤝🙌

🤩 Anything-3D-Objects
🔥 Anything-3DNovel-View
🥳 Anything-NeRF
😎 Any-3DFace
💘 Acknowledgements
Citation

🤩 Anything-3D-Objects

In this section, we showcase the results of combining Segment Anything with 3DFuse to segment and reconstruct 3D objects in the wild. Check out the following table for segmentation results and the corresponding 3D object:

Segmentation	Result

🔥 Anything-3DNovel-View

In this section, we demonstrate the combination of Segment Anything with Zero 1-to-3 to generate novel views of 3D objects. Check out the following images:

🥳 Anything-NeRF

In this section, we showcase the integration of Segment Anything with NeRF to generate new perspectives of objects set against intricate backgrounds. When an object is positioned in front of a plain, perspective-less background, NeRF typically struggles to reconstruct the scene. However, by eliminating the background, we can enhance NeRF's performance and facilitate more accurate reconstructions of scenes with objects presented in novel views.

Segmentation-1	Segmentation-2	Result

😎 Any-3DFace

In this section, we showcase the results of combining Segment Anything with HRN for accurate and detailed face reconstruction from in-the-wild images. Check out the following table for segmentation results and the corresponding face reconstruction:

Segmentation	Result

💘 Acknowledgements

We would like to acknowledge the following projects for their valuable contributions:

Citation

If you find this project helpful for your research, please consider citing the following BibTeX entry.

@misc{shen2023anything3d,
    title={Anything-3D: Towards Single-view Anything Reconstruction in the Wild}, 
    author={Qiuhong Shen and Xingyi Yang and Xinchao Wang},
    year={2023},
    eprint={2304.10261},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

And other projects

@article{kirillov2023segany,
    title={Segment Anything}, 
    author={Kirillov, Alexander and Mintun, Eric and Ravi, Nikhila and Mao, Hanzi and Rolland, Chloe and Gustafson, Laura and Xiao, Tete and Whitehead, Spencer and Berg, Alexander C. and Lo, Wan-Yen and Doll{\'a}r, Piotr and Girshick, Ross},
    journal={arXiv:2304.02643},
    year={2023}
}
@misc{liu2023zero1to3,
    title={Zero-1-to-3: Zero-shot One Image to 3D Object}, 
    author={Ruoshi Liu and Rundi Wu and Basile Van Hoorick and Pavel Tokmakov and Sergey Zakharov and Carl Vondrick},
    year={2023},
    eprint={2303.11328},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}
@inproceedings{Lei2023AHR,
    title={A Hierarchical Representation Network for Accurate and Detailed Face Reconstruction from In-The-Wild Images},
    author={Biwen Lei and Jianqiang Ren and Mengyang Feng and Miaomiao Cui and Xuansong Xie},
    year={2023}
}
@article{seo2023let,
    title={Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generation},
    author={Seo, Junyoung and Jang, Wooseok and Kwak, Min-Seop and Ko, Jaehoon and Kim, Hyeonsu and Kim, Junho and Kim, Jin-Hwa and Lee, Jiyoung and Kim, Seungryong},
    journal={arXiv preprint arXiv:2303.07937},
    year={2023}
}
@inproceedings{mildenhall2020nerf,
    title={NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis},
    author={Ben Mildenhall and Pratul P. Srinivasan and Matthew Tancik and Jonathan T. Barron and Ravi Ramamoorthi and Ren Ng},
    year={2020},
    booktitle={ECCV},
}

anything-3d's People

Contributors

Stargazers

Watchers

Forkers

haorand florinshen joranwang popcorn-attack gaohuan2015 hs0652 zergratino dq-soulie ithink3iam zhangzw12319 mfkiwl mzkaramat ydk1234 0iui0 machine-learning-headquarters miladshakerdn littletownstudio dylangh bingtian88 goyallon f901107 nitishymtpl qianqian121 oggyfaker algorithmlover2016 maguire1999 moileehyeji simboyz meronshuta chenyang48671155 yuanli12139 patonsun hanqiaojian angelababyhuang donnieburgs lidarsensorman wenhaoxiaozhang z99205388 0xrjman maisonde3cochons hufeihu jerryzfc 8giaduoxula shaoguangcheng jhchance dr-data jabrail-chumakov ilully0802 crinai haoyaogang primesho sorcererq tfgbestneal know-it-all-marketing-llc castcomzgeomi chunyu-lin-bjtu lidi100 cvcuiwei liangzuan1983 joncv shiyeli timwahoo tinfeijun aiot-creater luyao2001 5l1v3r1 wh-forker kellhuang ruilvdotcomceo ruilvcomceo bizai3000 ahhhz975 hengle averyyyyyy

anything-3d's Issues

Any Face 3d

Hello. Thank you for your work. Höw can I use Any face 3d?

大佬，具体实现开源么

如题，感谢大佬！

Suggestion - Integrate MobileSAM into the pipeline for lightweight and faster inference

Reference: https://github.com/ChaoningZhang/MobileSAM

Our project performs on par with the original SAM and keeps exactly the same pipeline as the original SAM except for a change on the image encode, therefore, it is easy to Integrate into any project.

MobileSAM is around 60 times smaller and around 50 times faster than original SAM, and it is around 7 times smaller and around 5 times faster than the concurrent FastSAM. The comparison of the whole pipeline is summarzed as follows:

Best Wishes,

Qiao

安装失败

安装Anything-3D/AnyObject3D时，通过pip install -r requirements.txt安装失败（主要是pytorch3d和basicsr），是否方便重新生成一份新的requirements.txt，并告知python的版本。谢谢

HRN demo

Hi, amazing project. However, the code of HRN is not released yet, how did you test it?

如何使用呢

您好，感谢您的工作作出的贡献。我有两个问题想请教您：
1 所选的目标是不限的吗？
2 我应该如何使用您的项目在自己的计算机上进行推理呢？

How to use the code?

Code Release

Hi @VainF @Adamdad @yu-rp @FlorinShum @OliverRensu ,
Thanks for the cool work of combining sam and 3Dfuse models.I am eagerly looking to test the demo on open source videos.
How soon the demo code can be released?

why？Can't load tokenizer for 'bert-base-uncased'.

OSError: Can't load tokenizer for 'bert-base-uncased'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'bert-base-uncased' is the correct path to a directory containing all relevant files for a BertTokenizer tokenizer.

cd /path/to/Anything-3D/AnyObject3D/src
mkdir weights && cd weights 
wget https://huggingface.co/jyseo/3DFuse_weights/resolve/main/models/3DFuse_sparse_depth_injector.ckpt

Where does this weight work?

is this the one given to the sam_checkpoint in the main.py script?

sam_checkpoint = "./checkpoint/sam_vit_h_4b8939.pth"

otherwise where is this sam_vit_h_4b8939.pth

Appreciate :-)

can't load tokenizer

您好请问这个应该是如何处理

Could not find the code

Can u mention somewhere that where is your main file