Giter Club home page Giter Club logo

blockfusion's Introduction

BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation

ACM Transactions on Graphics (SIGGRAPH'24)

This repo contains the official implementation for the paper "BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation".

drawing

drawing

Installation

Pytorch3d is required for postprocess. Come to offical page for installation instructions.

We modified diffusers to adapt triplane structure. Users should build based on the ./src

# clone this repo 
python setup.py install

Pretrained model

Pretrained weights of MLP, VAE, and Diffuser(Condition/Unconditioned). Download the model here [10GB] and extract to ./checkpoints.

Inference

To do uncondtional inference, run

python unconditioned_prediction.py --batch 4 --save output/uncond

To do single block conditional inference, run

python conditioned_prediction.py --layout samplelayouts/exp1_32-56-24/0_0.npy --save output/cond

We provide some sample layouts for demo. If you would like to draw your own layout maps, you can use drawtkinter.py to create your own layout. Here is a tutorial:

visualization

The unit of the graduations are in meters, furnitures need to be in reasonable scale. Walls should surround the floor and furniture should be placed on the floor.

The output layouts directory named after expname_xscale-zscale-stride, xscale-zscale-stride are given in decimeter.

Usage:

python drawtkinter.py --h 56 --w 56 --expname draw

Full pipeline(conditional prediction + extrapolation + resample) is contained in fullpipeline.py. Before running, specifying the layout directory.

python fullpipeline.py --layout samplelayouts/exp2_32-56-24 --resample 15 --save output

Citation

If you find our code or paper helps, please consider citing:

@article{blockfusion,
  title={BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation},
  author={Wu, Zhennan and Li, Yang and Yan, Han and Shang, Taizhang and Sun, Weixuan and Wang, Senbo and Cui, Ruikai and Liu, Weizhe and Sato, Hiroyuki and Li, Hongdong and Ji, Pan},
  journal={ACM Transactions on Graphics},
  volume={43},
  number={4},
  year={2024},
  doi={10.1145/3658188}
 }      

blockfusion's People

Contributors

rabbityl avatar

Stargazers

sulleyyy avatar Tianyu Huang avatar Flank Yang avatar sekkit avatar Baixin XU avatar Xiyue Wang avatar sophytoeat avatar Inhee Lee avatar  avatar  avatar Chengzeng avatar cvhadessun avatar Haipeng Wang avatar David avatar Wang Shuheng avatar  avatar Dongyu Yan avatar  avatar Quentin Leboutet avatar liuliuliuliu avatar Harsha Mohan avatar Beniko_J avatar 404Unkown avatar Jeff Carpenter avatar Jingwei Xu avatar Han Xu avatar Kaiwen Tu avatar 离离 avatar Kai avatar Franklin_Z avatar Zhe YAN avatar Yifan Lu avatar  avatar Tongyan Hua avatar Hyejin Mun avatar  avatar  avatar Hengyu MENG avatar Licheng Shen avatar Hyunggi Chang avatar Jiadai Sun avatar Jaeseok Park avatar Juwon Kim avatar  avatar  avatar Xiong Lin avatar Hyeontae Son avatar Hongrui Cai avatar Jiahao Shao avatar sumyyyyy avatar Qingcheng Zhao avatar Honghao Xu avatar Comar avatar  avatar Leheng Li avatar  avatar Yunhan Yang avatar Mahmud Ahmed avatar  avatar ROHAN KAR avatar yqdch avatar shaomq avatar Fan-Yun Sun avatar yoookon avatar YoungChan avatar Youquan Liu avatar  avatar Diankun Wu avatar  avatar Nobuyuki Umetani avatar Jin-Chuan Shi avatar  avatar Adeer Khan avatar gzg avatar Ernie Chu avatar Pan Ji avatar  avatar Haoxiang Guo avatar Lu Ming avatar bitm avatar Songqi Kong avatar Jaeah Lee avatar Fuqiang Zhao avatar  avatar Hong Yu avatar Yeshwanth Kumar avatar Jinmo Kim avatar  avatar Jeon Youngwoo avatar  avatar SereinH avatar CuiRuikai avatar Changjiang Cai avatar Shareef Ifthekhar avatar  avatar  avatar 邓旭东 avatar  avatar

Watchers

 avatar

blockfusion's Issues

about the grad of sdf model

hi,
thanks for your great work.
Is it convenient for you to share the code for calculating the SDF gradient? I have implemented the Eq.6 for calculating gradients in your article, but the results are always incorrect

issues with linear attention mode when running the main

Get errors below when running the main function:

constructing SpatialTransformer_ of depth 1 w/ 512 channels and 16 heads
Attention mode 'linear' is not available. Falling back to native attention. This is not a problem in Pytorch >= 2.0. FYI, you are running with PyTorch version 2.4.0

Triplane fitting MLP pretrained weight

Hello,
Thanks for the impressive work!

I’m trying to test the triplane fitting code located in the fit_triplane directory. However, it seems that the provided pretrained weights do not include the mlp.tar file. Could you provide an alternative link for the pretrained MLP weights(mlp.tar)?

Any practical installation guide?

Would you mind offer a enviroment version list you are using?
I've tried py39, pytorch 2.3.1 + cu118, build pytorch3d from source code, and build the custom diffusers according to your setup.py script.
And during executing the command: python unconditioned_prediction.py --batch 4 --save output/uncond ,got a lot of errors about missing modules like open3d, einops, xformers, etc... after I pip install them each by each. Still got this error:

undefined symbol: _ZN3c1015SmallVectorBaseIjE8grow_podEPvmm

It could be painful to guess the correct runtime enviroment...

FYI, I thinks its a matter of xformers version

link for pretrained weights

Thanks for the awesome work! The link for pretrained weights seem to be wrong, is there an alternative link?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.