LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes

*Jaeyoung Chung, *Suyoung Lee, Hyeongjin Nam, Jaerin Lee, Kyoung Mu Lee

• Linux: Ubuntu>18.04

conda create -n lucid python=3.9
conda activate lucid
pip install peft diffusers scipy numpy imageio[ffmpeg] opencv-python Pillow open3d torchvision gradio
pip install torch==2.0.1 timm==0.6.7 # ZoeDepth
pip install plyfile==0.8.1 # Gaussian splatting

cd submodules/depth-diff-gaussian-rasterization-min
python setup.py install
cd ../simple-knn
python setup.py install
cd ../..

# Default Example
python run.py

To run with your own inputs and prompts, attach following arguments after run.py.

• -img : path of input image.
• -t : text prompt. Can be either path to txt file or the text iteslf.
• -nt : negative text prompt. Can be either path to txt file or the text iteslf.
• -cg : camera extrinsic path for generating scenes. Can be one of "Rotate_360", "LookAround", or "LookDown".
• -cr : camera extrinsic path for rendering videos. Can be one of "Back_and_forth", "LLFF", or "Headbanging".
• --seed : manual seed for Stable Diffusion inpainting.
• --diff_steps : number of denoising steps for Stable Diffusion inpainting. Default is 50.
• -s : path to save results.

• ✅ December 7, 2023: Colab implementation is now available thanks to @camenduru!
• ✅ December 6, 2023: Code release!
• ✅ November 22, 2023: We have released our paper, LucidDreamer on arXiv.

Please cite us if you find our project useful!

@article{chung2023luciddreamer,
    title={LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes},
    author={Chung, Jaeyoung and Lee, Suyoung and Nam, Hyeongjin and Lee, Jaerin and Lee, Kyoung Mu},
    journal={arXiv preprint arXiv:2311.13384},
    year={2023}
}

We deeply appreciate ZoeDepth, Stability AI, and Runway for their models.

If you have any questions, please email [email protected], [email protected].