tencentarc / instantmesh Goto Github PK

if you try to run this on Windows, you might need to install the following dependencies manually:

1. conda install -c conda-forge cudatoolkit-dev -y
2. pip install Ninja
3. https://visualstudio.microsoft.com/visual-cpp-build-tools/
   and install c++ build tools with
   

after installed these additional dependencies, I could run it on my local with 3090

Great Job!

The huggingface demo link in the readme is currently hitting a 404. It may be a permissions issue.

As an aside, these results are simply phenomenal. Thank you for this awesome work.

I keep getting this error but couldnt find any solutions, everything is in the place, maybe i should add something to env variables? Or install something so it will work

(instantmesh) C:\Users\k1ckm3\InstantMesh>python run.py configs/instant-mesh-large.yaml examples/ver.png --save_video --export_texmap
Seed set to 42
Loading diffusion model ...
Loading pipeline components...: 0%| | 0/8 [00:00<?, ?it/s]The config attributes {'dropout': 0.0, 'reverse_transformer_layers_per_block': None} were passed to UNet2DConditionModel, but are not expected and will be ignored. Please verify your config.json configuration file.
Loading pipeline components...: 100%|█████████████████████████| 8/8 [00:06<00:00, 1.25it/s]
Loading custom white-background unet ...
Loading reconstruction model ...
Some weights of ViTModel were not initialized from the model checkpoint at facebook/dino-vitb16 and are newly initialized: ['encoder.layer.11.adaLN_modulation.1.bias', 'encoder.layer.0.adaLN_modulation.1.bias', 'encoder.layer.4.adaLN_modulation.1.weight', 'encoder.layer.5.adaLN_modulation.1.bias', 'encoder.layer.2.adaLN_modulation.1.bias', 'encoder.layer.6.adaLN_modulation.1.bias', 'encoder.layer.9.adaLN_modulation.1.weight', 'encoder.layer.10.adaLN_modulation.1.weight', 'encoder.layer.3.adaLN_modulation.1.weight', 'encoder.layer.2.adaLN_modulation.1.weight', 'encoder.layer.1.adaLN_modulation.1.bias', 'encoder.layer.4.adaLN_modulation.1.bias', 'encoder.layer.1.adaLN_modulation.1.weight', 'encoder.layer.8.adaLN_modulation.1.bias', 'encoder.layer.5.adaLN_modulation.1.weight', 'encoder.layer.10.adaLN_modulation.1.bias', 'encoder.layer.9.adaLN_modulation.1.bias', 'encoder.layer.11.adaLN_modulation.1.weight', 'encoder.layer.8.adaLN_modulation.1.weight', 'encoder.layer.3.adaLN_modulation.1.bias', 'encoder.layer.7.adaLN_modulation.1.weight', 'encoder.layer.0.adaLN_modulation.1.weight', 'encoder.layer.7.adaLN_modulation.1.bias', 'encoder.layer.6.adaLN_modulation.1.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using C:\Users\k1ckm3\AppData\Local\torch_extensions\torch_extensions\Cache\py310_cu121 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\k1ckm3\AppData\Local\torch_extensions\torch_extensions\Cache\py310_cu121\nvdiffrast_plugin\build.ninja...
Building extension module nvdiffrast_plugin...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module nvdiffrast_plugin...
Traceback (most recent call last):
File "C:\Users\k1ckm3\InstantMesh\run.py", line 132, in
model.init_flexicubes_geometry(device, fovy=30.0)
File "C:\Users\k1ckm3\InstantMesh\src\models\lrm_mesh.py", line 79, in init_flexicubes_geometry
renderer = NeuralRender(device, camera_model=camera)
File "C:\Users\k1ckm3\InstantMesh\src\models\geometry\render\neural_render.py", line 73, in init
self.ctx = dr.RasterizeCudaContext(device=device)
File "C:\Users\k1ckm3\anaconda3\envs\instantmesh\lib\site-packages\nvdiffrast\torch\ops.py", line 177, in init
self.cpp_wrapper = _get_plugin().RasterizeCRStateWrapper(cuda_device_idx)
File "C:\Users\k1ckm3\anaconda3\envs\instantmesh\lib\site-packages\nvdiffrast\torch\ops.py", line 118, in _get_plugin
torch.utils.cpp_extension.load(name=plugin_name, sources=source_paths, extra_cflags=opts, extra_cuda_cflags=opts+['-lineinfo'], extra_ldflags=ldflags, with_cuda=True, verbose=True)
File "C:\Users\k1ckm3\anaconda3\envs\instantmesh\lib\site-packages\torch\utils\cpp_extension.py", line 1308, in load
return _jit_compile(
File "C:\Users\k1ckm3\anaconda3\envs\instantmesh\lib\site-packages\torch\utils\cpp_extension.py", line 1736, in _jit_compile
return _import_module_from_library(name, build_directory, is_python_module)
File "C:\Users\k1ckm3\anaconda3\envs\instantmesh\lib\site-packages\torch\utils\cpp_extension.py", line 2136, in _import_module_from_library
module = importlib.util.module_from_spec(spec)
File "", line 571, in module_from_spec
File "", line 1176, in create_module
File "", line 241, in _call_with_frames_removed
ImportError: DLL load failed while importing nvdiffrast_plugin: The specified module cannot be found.

raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/sudo-ai/zero123
plus-v1.2 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbaceffa620>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), '(Request ID: ca57e223-bc50-41fd-be5a-67ccfba32a47)')

Hi, thanks for the great work. I've read the code. And found that the depth loss was calculated, yet not sum into the total loss. Is there any reasons for doing so?

Hi, this is truly amazing work!

We'd like to run it on a LAN of multiple servers, which have 8 -24gm vram unevenly. Is it possible to run this model on multiple GPUs? It looks like most people are running it on 24gb vram as a bare minimum

Thanks

Hello! Let me first say congrats and thanks for all your work, InstantMesh is absolutely amazing!

I am wondering about the possibility of aligning the initial input image with the produced mesh, specifically extracting the camera pose of the input image which could then be applied to the mesh such that it "looks the same". I know that One-2-3-45 has this capability (code here), and one of their multi-view diffusion-created images is a recreated version of the initial image at a guessed pose. I also believe that One-2-3-45 builds their poses relative to this initial image's pose calculation.

It seems like Zero123++ (which InstantMesh uses as the multi-view diffusion model) utilizes only the content of the initial image and doesn't bother figuring out a decent "absolute" pose of the initial image, but I haven't looked too closely into the Zero123++ code quite yet.

If InstantMesh doesn't immediately have this capability, could you point me in the general direction of a solution? Use the "elevation_estimate" as seen in One-2-3-45, with 0 degree azimuth (it looks like the 6 poses produced here are assuming initial azimuth being 0 degrees?)

Thanks for any help!

用examples下自带的图片，生成的效果较好。用自己的图片，效果不佳。请问需要对图片做什么特别处理吗？怎么提高泛化性？

Thanks for the amazing work. I noticed that after finetuning to white bg, the output image has a large scale:

Do you use different intrinsics when finetuning? Would large scale output better for the reconstruction stage?

怎么使得3D obj文件更加清晰

(instantmesh) me@myshit InstantMesh-main % python run.py configs/instant-mesh-large.yaml examples/hatsune_miku.png --save_video --no_rembg
Intel MKL WARNING: Support of Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) enabled only processors has been deprecated. Intel oneAPI Math Kernel Library 2025.0 will require Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.
Intel MKL WARNING: Support of Intel(R) Streaming SIMD Extensions 4.2 (Intel(R) SSE4.2) enabled only processors has been deprecated. Intel oneAPI Math Kernel Library 2025.0 will require Intel(R) Advanced Vector Extensions (Intel(R) AVX) instructions.
/Users/me/opt/anaconda3/envs/instantmesh/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
Seed set to 42
Loading diffusion model ...
Loading pipeline components...: 25%|████████████████████████▊ | 2/8 [00:00<00:01, 5.49it/s]The config attributes {'dropout': 0.0, 'reverse_transformer_layers_per_block': None} were passed to UNet2DConditionModel, but are not expected and will be ignored. Please verify your config.json configuration file.
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:06<00:00, 1.27it/s]
Loading custom white-background unet ...
Traceback (most recent call last):
File "/Users/me/Desktop/InstantMesh-main/run.py", line 109, in
state_dict = torch.load(infer_config.unet_path, map_location='cpu')
File "/Users/me/opt/anaconda3/envs/instantmesh/lib/python3.10/site-packages/torch/serialization.py", line 998, in load
with _open_file_like(f, 'rb') as opened_file:
File "/Users/me/opt/anaconda3/envs/instantmesh/lib/python3.10/site-packages/torch/serialization.py", line 445, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/Users/me/opt/anaconda3/envs/instantmesh/lib/python3.10/site-packages/torch/serialization.py", line 426, in init
super().init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'ckpts/diffusion_pytorch_model.bin'

Hello is it possible to add InstantMesh as extension in Automatic1111 WebUi ?

我用24G的显存，报错了，是不是显存不够？

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 512.00 MiB. GPU 0 has a total capacty of 23.69 GiB of which 368.94 MiB is free. Including non-PyTorch memory, this process has 23.33 GiB memory in use. Of the allocated memory 22.72 GiB is allocated by PyTorch, and 287.94 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Hi, thanks for the great work. I'm wondering if you could provide
(1) the evaluation data list for GSO and Omni3D.
(2) what's your rendering configuration?
(3) For the camera setting, are the cameras always facing towards the sphere center?
Thanks in advance for your help!

I implemented ComfyUI support for this, https://github.com/jtydhr88/ComfyUI-InstantMesh

执行python app.py后下载的文件放在哪里了，怎么删除啊，内存告急😅

I found that "demo.launch(server_name="0.0.0.0", server_port=43839)" in app.py makes that when you open the url, it makes the app don't work, but there's a solution, changing "server_name" and "server_port":

Example:

demo.launch(server_name="127.0.0.1", server_port=7860)

I am getting a CUDA out-of-memory error when running this on Ubuntu 22.04.04 with two RXT 3090 GPUs (2x24GB VRAM).

The error:

/home/user/InstantMesh/app.py:135: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:206.)
show_image = torch.from_numpy(show_image) # (960, 640, 3)
/tmp/tmpywepamk4.obj
0%| | 0/6 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/user/miniconda3/envs/instantmesh/lib/python3.10/site-packages/gradio/routes.py", line 488, in run_predict
output = await app.get_blocks().process_api(
File "/home/user/miniconda3/envs/instantmesh/lib/python3.10/site-packages/gradio/blocks.py", line 1431, in process_api
result = await self.call_function(
File "/home/user/miniconda3/envs/instantmesh/lib/python3.10/site-packages/gradio/blocks.py", line 1103, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/user/.local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/user/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/home/user/.local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/home/user/miniconda3/envs/instantmesh/lib/python3.10/site-packages/gradio/utils.py", line 707, in wrapper
response = f(*args, **kwargs)
File "/home/user/InstantMesh/app.py", line 199, in make3d
frame = model.forward_geometry(
File "/home/user/InstantMesh/src/models/lrm_mesh.py", line 280, in forward_geometry
mesh_v, mesh_f, sdf, deformation, v_deformed, sdf_reg_loss = self.get_geometry_prediction(planes)
File "/home/user/InstantMesh/src/models/lrm_mesh.py", line 165, in get_geometry_prediction
sdf, deformation, sdf_reg_loss, weight = self.get_sdf_deformation_prediction(planes)
File "/home/user/InstantMesh/src/models/lrm_mesh.py", line 110, in get_sdf_deformation_prediction
sdf, deformation, weight = torch.utils.checkpoint.checkpoint(
File "/home/user/miniconda3/envs/instantmesh/lib/python3.10/site-packages/torch/_compile.py", line 24, in inner
return torch._dynamo.disable(fn, recursive)(*args, **kwargs)
File "/home/user/miniconda3/envs/instantmesh/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn
return fn(*args, **kwargs)
File "/home/user/miniconda3/envs/instantmesh/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner
return fn(*args, **kwargs)
File "/home/user/miniconda3/envs/instantmesh/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 458, in checkpoint
ret = function(*args, **kwargs)
File "/home/user/InstantMesh/src/models/renderer/synthesizer_mesh.py", line 132, in get_geometry_prediction
sdf, deformation, weight = self.decoder.get_geometry_prediction(sampled_features, flexicubes_indices)
File "/home/user/InstantMesh/src/models/renderer/synthesizer_mesh.py", line 76, in get_geometry_prediction
grid_features = torch.index_select(input=sampled_features, index=flexicubes_indices.reshape(-1), dim=1)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 15.00 GiB. GPU 0 has a total capacty of 23.66 GiB of which 14.47 GiB is free. Process 1759 has 254.00 MiB memory in use. Including non-PyTorch memory, this process has 8.32 GiB memory in use. Of the allocated memory 7.23 GiB is allocated by PyTorch, and 288.87 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Hello @JustinPack @yxgeee @bluestyle97 @eltociear I am getting cuda out of memory error when trying to run this model on my device using command line. I have 2GPU's each of 16GB. Issue here is that it is using only 1 GPU and its not accessing the second one. How can I make the pipeline use multiple GPU's? I ran run.py script using command line without rmb. Still I am getting the issue.
Command used: python run.py configs/instant-mesh-large.yaml examples/hatsune_miku.png --save_video

Error:
Traceback (most recent call last): File "/home/jupyter/InstantMesh/run.py", line 221, in <module> mesh_out = model.extract_mesh( File "/home/jupyter/InstantMesh/src/models/lrm_mesh.py", line 357, in extract_mesh mesh_v, mesh_f, sdf, deformation, v_deformed, sdf_reg_loss = self.get_geometry_prediction(planes) File "/home/jupyter/InstantMesh/src/models/lrm_mesh.py", line 165, in get_geometry_prediction sdf, deformation, sdf_reg_loss, weight = self.get_sdf_deformation_prediction(planes) File "/home/jupyter/InstantMesh/src/models/lrm_mesh.py", line 110, in get_sdf_deformation_prediction sdf, deformation, weight = torch.utils.checkpoint.checkpoint( File "/opt/conda/envs/instantmesh/lib/python3.10/site-packages/torch/_compile.py", line 24, in inner return torch._dynamo.disable(fn, recursive)(*args, **kwargs) File "/opt/conda/envs/instantmesh/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 328, in _fn return fn(*args, **kwargs) File "/opt/conda/envs/instantmesh/lib/python3.10/site-packages/torch/_dynamo/external_utils.py", line 17, in inner return fn(*args, **kwargs) File "/opt/conda/envs/instantmesh/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 458, in checkpoint ret = function(*args, **kwargs) File "/home/jupyter/InstantMesh/src/models/renderer/synthesizer_mesh.py", line 132, in get_geometry_prediction sdf, deformation, weight = self.decoder.get_geometry_prediction(sampled_features, flexicubes_indices) File "/home/jupyter/InstantMesh/src/models/renderer/synthesizer_mesh.py", line 76, in get_geometry_prediction grid_features = torch.index_select(input=sampled_features, index=flexicubes_indices.reshape(-1), dim=1) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 15.00 GiB. GPU 0 has a total capacty of 14.58 GiB of which 10.58 GiB is free. Including non-PyTorch memory, this process has 3.99 GiB memory in use. Of the allocated memory 3.63 GiB is allocated by PyTorch, and 226.96 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

启动需要多少现存

Thanks for the project ❤️ I made a jupyter notebook and replicate demo 🥳 I hope you like it.

https://github.com/camenduru/InstantMesh-jupyter
https://replicate.com/camenduru/instantmesh

Traceback (most recent call last):
File "/InstantMesh/run.py", line 213, in
planes = model.forward_planes(images, input_cameras)
File "/InstantMesh/src/models/lrm_mesh.py", line 98, in forward_planes
planes = self.transformer(image_feats)
File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/InstantMesh/src/models/decoder/transformer.py", line 115, in forward
x = layer(x, image_feats)
File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/InstantMesh/src/models/decoder/transformer.py", line 65, in forward
x = x + self.self_attn(before_sa, before_sa, before_sa)[0]
File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/py310/lib/python3.10/site-packages/torch/nn/modules/activation.py", line 1144, in forward
return torch._native_multi_head_attention(
TypeError: _native_multi_head_attention(): argument 'qkv_bias' (position 7) must be Tensor, not NoneType

Do you know what's wrong with this problem? can anyone help me?

On Windows, installation following the guide requires some additional steps such as ensuring Ninja is installed and ensuring you have CUDA installed from Nvidia.

Additionally, Triton is not available for Windows via the standard pip install triton approach, but can be installed by utilizing a prebuilt one available from a huggingface repo like so.

pip install https://huggingface.co/r4ziel/xformers_pre_built/resolve/main/triton-2.0.0-cp310-cp310-win_amd64.whl

Finally, on windows, it seems the removal of line 321 in the app.py file via commenting out width=768 is required due to depreciation. I was unable to launch the gradio instance until I had done as much.

After installation I am getting these errors. How to resolve it please? Thanks

Surprised to find that generating the obj + texture (https://replicate.com/p/r366c4nc49rga0cexcr8x5ys6r @ 35s) takes longer than obj + texture + video (https://replicate.com/p/z3s1hyrmtxrga0cexcr8nw89d8 @ 25s)

Please optimize texture saving so that save_obj_with_mtl is faster than generating the video

--

Benchmarks:
obj only @ 12.4s https://replicate.com/p/2ev8akz2s5rga0cexkmvzxg064
obj + video 13.1s @ https://replicate.com/p/djxg387xmhrga0cexkmtbxj89c
obj + texture @ 34.3s https://replicate.com/p/r366c4nc49rga0cexcr8x5ys6r
(and why does this one take less time, when more assets are generated?)
obj + texture + video @ 25.8s https://replicate.com/p/z3s1hyrmtxrga0cexcr8nw89d8

Here's the complete benchmark table based on parameters -
The input image is 2048x2048

The input image is 512x512

GOOD JOB! Why change the absolute elevations of 30° and -20° in zero123++ to absolute elevations of 20° and -10°?"

Hi, when I tried to run the code, I received the following issue. I think the problem comes from here:

pipeline = DiffusionPipeline.from_pretrained(
"sudo-ai/zero123plus-v1.2",
custom_pipeline="zero123plus",
torch_dtype=torch.float16,
cache="/scratch/huggingface"
)
Do you know if it is a problem linked with InstantMesh or Huggingface?

Could not locate the pipeline.py inside zero123plus.
Traceback (most recent call last):
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
response.raise_for_status()
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://raw.githubusercontent.com/huggingface/diffusers/v0.27.2/examples/community/zero123plus.py

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/scratch2/yuxili/interiorDesign/InstantMesh/run_instantmesh.py", line 98, in
pipeline = DiffusionPipeline.from_pretrained(
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
return fn(*args, **kwargs)
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 671, in from_pretrained
cached_folder = cls.download(
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
return fn(*args, **kwargs)
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1317, in download
pipeline_class = _get_pipeline_class(
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/diffusers/pipelines/pipeline_loading_utils.py", line 323, in _get_pipeline_class
return get_class_from_dynamic_module(
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
return fn(*args, **kwargs)
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/diffusers/utils/dynamic_modules_utils.py", line 441, in get_class_from_dynamic_module
final_module = get_cached_module_file(
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
return fn(*args, **kwargs)
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/diffusers/utils/dynamic_modules_utils.py", line 287, in get_cached_module_file
resolved_module_file = cached_download(
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn
return fn(*args, **kwargs)
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 697, in cached_download
r = _request_wrapper(
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 369, in _request_wrapper
response = _request_wrapper(
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 393, in _request_wrapper
hf_raise_for_status(response)
File "/scratch2/yuxili/miniconda3/envs/env_test_test/lib/python3.10/site-packages/huggingface_hub/utils/_errors.py", line 371, in hf_raise_for_status
raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 404 Client Error: Not Found for url: https://raw.githubusercontent.com/huggingface/diffusers/v0.27.2/examples/community/zero123plus.py

imagemesh_2gpu.txt

the attached python scripts splits functions across two gpus. allows for running the program on 2x3090s simutaneously with stable diffusion.

I'm sure I'm doing this whole process wrong, I am not a programmer, but I got Claude3 to output the code, and it works flawlessly.

Enjoy.

requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/sudo-ai/zero123plus-v1.2 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fa9a8358210>, 'Connection to huggingface.co timed out. (connect timeout=None)'))"), '(Request ID: ae9ede8d-b9e4-4530-bc6a-7e259b76257e)')

请问在huggingface的demo上使用的哪个模型呢？我在A5000上使用instant-nerf-large进行推理，用时是huggingface上的两倍，而且生成质量不如huggingface的demo。

Hi, thanks for the great work!

I wonder if it would be possible to modify the code in order to specify the (x, y, z) dimensions of the generated 3D model, in a similar sense as specifying image height and width for text-to-image models?

Hello! When running the model I am recieving a DLL load error.

ImportError: DLL load failed while importing nvdiffrast_plugin: The specified module could not be found.

The entire log is as follows:

PS E:\instantmesh> python run.py configs/instant-mesh-large.yaml examples/hatsune_miku.png --save_video --export_texmap
Seed set to 42
Loading diffusion model ...
Loading pipeline components...:  12%|██████▌                                             | 1/8 [00:00<00:02,  2.96it/s]The config attributes {'dropout': 0.0, 'reverse_transformer_layers_per_block': None} were passed to UNet2DConditionModel, but are not expected and will be ignored. Please verify your config.json configuration file.
Loading pipeline components...: 100%|████████████████████████████████████████████████████| 8/8 [00:08<00:00,  1.10s/it]
Loading custom white-background unet ...
Loading reconstruction model ...
Some weights of ViTModel were not initialized from the model checkpoint at facebook/dino-vitb16 and are newly initialized: ['encoder.layer.5.adaLN_modulation.1.weight', 'encoder.layer.7.adaLN_modulation.1.weight', 'encoder.layer.4.adaLN_modulation.1.weight', 'encoder.layer.1.adaLN_modulation.1.bias', 'encoder.layer.10.adaLN_modulation.1.weight', 'encoder.layer.6.adaLN_modulation.1.weight', 'encoder.layer.5.adaLN_modulation.1.bias', 'encoder.layer.0.adaLN_modulation.1.bias', 'encoder.layer.4.adaLN_modulation.1.bias', 'encoder.layer.9.adaLN_modulation.1.weight', 'encoder.layer.3.adaLN_modulation.1.weight', 'encoder.layer.2.adaLN_modulation.1.weight', 'encoder.layer.10.adaLN_modulation.1.bias', 'encoder.layer.3.adaLN_modulation.1.bias', 'encoder.layer.11.adaLN_modulation.1.bias', 'encoder.layer.2.adaLN_modulation.1.bias', 'encoder.layer.1.adaLN_modulation.1.weight', 'encoder.layer.0.adaLN_modulation.1.weight', 'encoder.layer.9.adaLN_modulation.1.bias', 'encoder.layer.6.adaLN_modulation.1.bias', 'encoder.layer.11.adaLN_modulation.1.weight', 'encoder.layer.8.adaLN_modulation.1.bias', 'encoder.layer.8.adaLN_modulation.1.weight', 'encoder.layer.7.adaLN_modulation.1.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
  File "E:\instantmesh\run.py", line 132, in <module>
    model.init_flexicubes_geometry(device, fovy=30.0)
  File "E:\instantmesh\src\models\lrm_mesh.py", line 79, in init_flexicubes_geometry
    renderer = NeuralRender(device, camera_model=camera)
  File "E:\instantmesh\src\models\geometry\render\neural_render.py", line 73, in __init__
    self.ctx = dr.RasterizeCudaContext(device=device)
  File "C:\Users\Weston\AppData\Local\Programs\Python\Python310\lib\site-packages\nvdiffrast\torch\ops.py", line 177, in __init__
    self.cpp_wrapper = _get_plugin().RasterizeCRStateWrapper(cuda_device_idx)
  File "C:\Users\Weston\AppData\Local\Programs\Python\Python310\lib\site-packages\nvdiffrast\torch\ops.py", line 118, in _get_plugin
    torch.utils.cpp_extension.load(name=plugin_name, sources=source_paths, extra_cflags=opts, extra_cuda_cflags=opts+['-lineinfo'], extra_ldflags=ldflags, with_cuda=True, verbose=False)
  File "C:\Users\Weston\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 1308, in load
    return _jit_compile(
  File "C:\Users\Weston\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 1736, in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
  File "C:\Users\Weston\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 2136, in _import_module_from_library
    module = importlib.util.module_from_spec(spec)
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1176, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: DLL load failed while importing nvdiffrast_plugin: The specified module could not be found.

Any ideas how this could happen? Did I miss a porgram that must be installed?

It seems that objects predominantly colored in white, such as toilet paper, are not detected effectively, possibly due to the model being trained primarily with images featuring white backgrounds. Conversely, the model performs well with objects that are not primarily white. Could you suggest any potential solutions to address this issue?

Thanks for your great work!

I wonder how you select the range of elevations and azimuths when rendering the filtered objaverse. This may influence the performance of the trained model.

I am looking forward to your reply!

Loading diffusion model ...
Fetching 16 files: 31%|███████████████████████▏ | 5/16 [00:00<00:01, 5.84it/s]
Traceback (most recent call last):
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
conn.connect()
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connection.py", line 653, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connection.py", line 806, in ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 465, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 509, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/ssl.py", line 1104, in _create
self.do_handshake()
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/ssl.py", line 1375, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/mk/.local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
retries = retries.increment(
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment
raise reraise(type(error), error, _stacktrace)
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/util/util.py", line 38, in reraise
raise value.with_traceback(tb)
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
response = self._make_request(
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
raise new_e
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
self._validate_conn(conn)
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
conn.connect()
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connection.py", line 653, in connect
sock_and_verified = _ssl_wrap_socket_and_match_hostname(
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/connection.py", line 806, in ssl_wrap_socket_and_match_hostname
ssl_sock = ssl_wrap_socket(
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 465, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
File "/home/mk/.local/lib/python3.10/site-packages/urllib3/util/ssl.py", line 509, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/ssl.py", line 1104, in _create
self.do_handshake()
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/ssl.py", line 1375, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/mk/InstantMesh-main/app.py", line 75, in
pipeline = DiffusionPipeline.from_pretrained(
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 932, in from_pretrained
cached_folder = cls.download(
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1507, in download
cached_folder = snapshot_download(
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 239, in snapshot_download
thread_map(
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 69, in thread_map
return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/tqdm/contrib/concurrent.py", line 51, in _executor_map
return list(tqdm_class(ex.map(fn, *iterables, chunksize=chunksize), **kwargs))
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/tqdm/std.py", line 1181, in iter
for obj in iterable:
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/concurrent/futures/_base.py", line 621, in result_iterator
yield _result_or_cancel(fs.pop())
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/concurrent/futures/_base.py", line 319, in _result_or_cancel
return fut.result(timeout)
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/concurrent/futures/_base.py", line 458, in result
return self.__get_result()
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 214, in _inner_hf_hub_download
return hf_hub_download(
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1431, in hf_hub_download
http_get(
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 515, in http_get
r = _request_wrapper(
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 452, in _request_wrapper
return http_backoff(
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 258, in http_backoff
response = session.request(method=method, url=url, **kwargs)
File "/home/mk/.local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/home/mk/.local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/home/mk/anaconda3/envs/InstantMesh/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 63, in send
return super().send(request, *args, **kwargs)
File "/home/mk/.local/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: (ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')), '(Request ID: 0a2fe285-3285-4aa0-8e75-80deb64f22d2)')

I noticed that the texture quality of my output did not meet my expectations, especially when compared to the demo results which looked significantly better.

Could you please help me understand which parameters might influence the texture quality? Additionally, I would like to know the specific parameters used in the demo for comparison.

Here are the default parameters I used:

model_config:
  target: src.models.lrm_mesh.InstantMesh
  params:
    encoder_feat_dim: 768
    encoder_freeze: false
    encoder_model_name: facebook/dino-vitb16
    transformer_dim: 1024
    transformer_layers: 16
    transformer_heads: 16
    triplane_low_res: 32
    triplane_high_res: 64
    triplane_dim: 80
    rendering_samples_per_ray: 64
    grid_res: 128
    grid_scale: 2.1


infer_config:
  unet_path: ckpts/diffusion_pytorch_model.bin
  model_path: ckpts/instant_mesh_large.ckpt
  texture_resolution: 1024
  render_resolution: 1024

Input image

Result

Any guidance or suggestions would be greatly appreciated!

Thank you for your time and assistance.

/home/zhangzhilong/tools/anaconda/envs/instantmesh/lib/python3.10/site-packages/torch/include/torch/csrc/profiler/util.h:34:10: 致命错误：x86intrin.h：No such file or directory
34 | #include <x86intrin.h>
| ^~~~~~~~~~~~~
编译中断。
ninja: build stopped: subcommand failed.

The generated 3D mode is blurry

执行python run.py configs/instant-mesh-large.yaml examples/hatsune_miku.png --save_video命令后
Loading pipeline components...: 0%| | 0/8 [00:00<?, ?it/s]The config attributes {'dropout': 0.0, 'reverse_transformer_layers_per_block': None} were passed to UNet2DConditionModel, but are not expected and will be ignored. Please verify your config.json configuration file.g (…)ch_model.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 3.46G/3.46G [01:25<00:00, 56.8MB/s]
Segmentation fault

Hello, thank you for your brilliant work on feed-forward-style Image-to-3D generation! I have tried your work on the HuggingFace demo and it works well!
I want to run your code on my local GPUs. However, due to some inevitable problems, my GPUs do not support the newest 12.x CUDA version and cannot be updated. So do you have any methods to run the code based on PyTorch 1.1x (such as 1.12, 1.13) and CUDA 11.x (such as 11.4, 11.6)?

Thanks a lot! :)

Hi, when starting app.py im getting the gradio error mentioned above.
Gradio version is 4.26.0

i resolved it by changing line 321 in app.py from
width=768
to
min_width=768

Hi @bluestyle97, I'm trying to train the model, but how do I set up the dataset, does the train.py download and preprocess the filtered objaverse dataset, or what do we have to do it, like it's giving an error:
No such file or directory: 'data/objaverse/filtered_obj_name.json'
Can you please provide this file?

Thanks for your great work!

I think that in the config file instant-mesh-large-train.yaml

I am looking forward to your reply! Thanks!

模型文件太大，希望放在文件存储上给程序读取，但是没有描述中说的 ckpts 文件

模型和代码文件欢迎发布到wisemodel.cn开源社区哦

it would be awesome if you could give her 10 pictures from different sides and she would make an exact 3D model out of it

感谢大佬的工作，这是我目前见到的最好的模型了。比triposr， openlrm, crm都要好很多，方法也很直截了当，先用mvd生成多视图，然后再接上transformer去猜triplane tokens，然后再接上flexicube组成一套differentiable的pipeline，直接去优化mesh。不过我实际infer的时候，发现mesh虽然学的不错，但是并没有很好的保留原有的纹理特征。请问可以在我的数据集上finetune解决这个问题吗？可以给一些指导意见吗？ 感谢