Comments (10)
Hi @wenqsun, can you please provide some guidance on how to set up the dataset properly?
from instantmesh.
Oh, I am just thinking about this issue. The elevations and azimuths of training data may influence the inference performance. For example, in this work, they use zero123++ to generate multi-view images, and when I use other multi-view diffusion models with different elevations and azimuths, the generated mesh somehow degrades.
Thus, I wonder how the authors select the range of camera pose for the training dataset, which may be related to the multi-view diffusion models you use during the inference stage. This will help me redesign the rendering strategy.
from instantmesh.
@wenqsun @kunalkathare We render our dataset following the setting of LRM. To be specific, each 3D model is normalized into a unit cube in the range of [-1, 1]. We render 32 views for each model, with camera distances in the range of [2.0, 3.5], camera heights (z axis) in the range of [-0.75, 1.6], fov=50.
Our reconstruction model is trained with free-viewpoint images, so its performance should be irrelevant to which multi-view diffusion model we use. Since you mentioned that "when I use other multi-view diffusion models with different elevations and azimuths, the generated mesh somehow degrades", could you please clarify which multi-view diffusion model are you using? Also, how did you set the corresponding input camera poses?
from instantmesh.
Hi @bluestyle97, I have used the blender script from LRM, now I have a folder with 32 poses, a folder with 32 rendered images and an intrinsic.npy file for each object.
Now I have created folders in the following order:
Instant mesh
- data
- objaverse
- input_image_folder
- target_image_folder
- filtered_obj_name.json
- valid_samples
Can you please tell me exactly what to put in these folders, and the format of the json file?
- objaverse
from instantmesh.
Hi @bluestyle97, I have used the blender script from LRM, now I have a folder with 32 poses, a folder with 32 rendered images and an intrinsic.npy file for each object.
Now I have created folders in the following order:
Instant mesh
- data
- objaverse
- input_image_folder
- target_image_folder
- filtered_obj_name.json
- valid_samples
Can you please tell me exactly what to put in these folders, and the format of the json file?
Hi @JustinPack , can you please provide some guidance or can you please share the script file used to render objaverse.
from instantmesh.
Hey @kunalkathare by LRM you means OpenLRM, right?
from instantmesh.
Hey @kunalkathare by LRM you means OpenLRM, right?
Yes
from instantmesh.
Hi @bluestyle97 - I had a look at the OpenLRM blender script but it does not include depth maps or normals. I have been using the following additional bpy
settings to render normal maps but they do not seem quite in line with what your InstantMesh model expects. Am I doing something wrong? This is done with Blender v4.0.0.
nodes = bpy.context.scene.node_tree.nodes
links = bpy.context.scene.node_tree.links
nodes.clear()
render_layers = nodes.new("CompositorNodeRLayers")
bpy.context.view_layer.use_pass_normal = True
# rescale [-1, 1] * 0.5 + 0.5 to [0, 1]
node_normal_scale = nodes.new(type="CompositorNodeMixRGB")
node_normal_scale.blend_type = "MULTIPLY"
node_normal_scale.inputs[2].default_value = (0.5, 0.5, 0.5, 1)
links.new(render_layers.outputs["Normal"], node_normal_scale.inputs[1])
node_normal_bias = nodes.new(type="CompositorNodeMixRGB")
node_normal_bias.blend_type = "ADD"
node_normal_bias.inputs[2].default_value = (0.5, 0.5, 0.5, 0)
links.new(node_normal_scale.outputs[0], node_normal_bias.inputs[1])
# save normals as png image
node_normal = nodes.new(type="CompositorNodeOutputFile")
node_normal.label = "Normal Output"
node_normal.base_path = "/"
node_normal.file_slots[0].use_node_format = True
node_normal.format.file_format = "PNG"
node_normal.format.color_mode = "RGBA"
links.new(node_normal_bias.outputs[0], node_normal.inputs[0])
from instantmesh.
@kunalkathare hello, I also have the LRM dataset, which is also rendered in blender. How can I use the training on instantmesh? Thank you very much for your help
from instantmesh.
Does any one of you know how to prepare the data for training? I referred openLRM blender script, will it be the same in the case of instant mesh?
from instantmesh.
Related Issues (20)
- GPU not used for diffusion?
- ninja: build stopped: subcommand failed.
- soldier
- 3 d image
- Support fbx export
- can supports texmap in run app.py ?
- Out of memory running the gradio example
- 训练配置
- How many epochs HOT 1
- Finish writing the rendering script !!!! Look!! HOT 3
- ninja: build stopped: subcommand failed. HOT 1
- Loading diffusion model ... Loading pipeline components...: 12% HOT 1
- how much gpu memory is required to train the model? HOT 1
- Some weights of ViTModel were not initialized from the model HOT 1
- Align the generated mesh with Zero123++'s coordinate system
- Train zero123 with own dataset
- 3D
- Where to place the fine-tuned weights? HOT 2
- Run Run. py to generate 6 different angle images with empty output images and pure black images. Guess it was a sampling error? There is no problem printing the input images. Could you please help me take a look, experts? Thank you HOT 1
- A question
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from instantmesh.