wu-cvgl / mvcontrol Goto Github PK
View Code? Open in Web Editor NEWOfficial implementation of "Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting"
Home Page: https://lizhiqi49.github.io/MVControl/
License: MIT License
Official implementation of "Controllable Text-to-3D Generation via Surface-Aligned Gaussian Splatting"
Home Page: https://lizhiqi49.github.io/MVControl/
License: MIT License
Can we achieve image-to-3D by changing the diffusion model in sds loss to zero1-to-3?
Hi, Congratulations to your wonderful work, which is very inspiring and helpful.
But after reading this paper, a question comes to me, I wonder with given conditions, can the MVControl handle text prompts that are composed of several items or contain some attaching properties or verbs? This is one prevailing issue in T2I generation.
Again, thank you for your great job.
Hello, thank you for your work on adding a ControlNet on MVDream. I am curious about using multi-view (4-view for MVDream) images instead of only one image as input for MVControlNet, since MVDream can handle multi-view images during the denoising process and ControlNet is a trainable copy of the architecture of MVDream.
If I want to fine-tune (MVDream + MVControlNet) to achieve multi-view X to multi-view Y, is the MVControl compatible to it?
Thank you!
个人感觉:
照片(rgb,rgba): 90%,草图(手工简单绘制): 10%,这两种是比较自然的输入形态。
为什么要将depth(难道是3D sensor),normal,canny, scribble这种作为输入呢?
Thanks
Hi, there is a problem on running the code:
python app_stage1.py big --resume pretrained/model_fp16.safetensors --condition_type $condition_type
/home/dubaiprince/Projects/MVControl-threestudio/extern/lgm/attention.py:22: UserWarning: xFormers is available (Attention)
warnings.warn("xFormers is available (Attention)")
[INFO] Loaded checkpoint from pretrained/model_fp16.safetensors
unet/diffusion_pytorch_model.safetensors not found
Loading pipeline components...: 17%|██ | 1/6 [00:00<00:00, 125.69it/s]
Traceback (most recent call last):
File "/home/dubaiprince/Projects/MVControl-threestudio/app_stage1.py", line 63, in <module>
pipe_mvcontrol = load_mvcontrol_pipeline(
File "/home/dubaiprince/Projects/MVControl-threestudio/extern/mvcontrol/pipeline_mvcontrol.py", line 966, in load_mvcontrol_pipeline
pipe = MVControlPipeline.from_pretrained(
File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/diffusers/pipelines/pipeline_utils.py", line 1093, in from_pretrained
loaded_sub_model = load_sub_model(
File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/diffusers/pipelines/pipeline_utils.py", line 386, in load_sub_model
class_obj, class_candidates = get_class_obj_and_candidates(
File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/site-packages/diffusers/pipelines/pipeline_utils.py", line 317, in get_class_obj_and_candidates
library = importlib.import_module(library_name)
File "/home/dubaiprince/miniconda3/envs/threestudio/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'camera_proj'
Can you suggest anyway to fix it?
File "/opt/anaconda3/envs/v3d/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3682, in _pad
encoded_inputs["attention_mask"] = encoded_inputs["attention_mask"] + [0] * difference
OverflowError: cannot fit 'int' into an index-sized integer
Traceback (most recent call last):
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/gradio/queueing.py", line 407, in call_prediction
output = await route_utils.call_process_api(
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/gradio/route_utils.py", line 226, in call_process_api
output = await app.get_blocks().process_api(
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/gradio/blocks.py", line 1550, in process_api
result = await self.call_function(
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/gradio/blocks.py", line 1185, in call_function
prediction = await anyio.to_thread.run_sync(
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 851, in run
result = context.run(func, *args)
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/gradio/utils.py", line 661, in wrapper
response = f(*args, **kwargs)
File "/home/dubaiprince/Projects/MVControl-threestudio/app_stage1.py", line 237, in process
image = model.gs.render(gaussians, cam_view.unsqueeze(0), cam_view_proj.unsqueeze(0), cam_pos.unsqueeze(0), scale_modifier=1)['image']
File "/home/dubaiprince/Projects/MVControl-threestudio/extern/lgm/gs.py", line 76, in render
rendered_image, radii, rendered_depth, rendered_alpha = rasterizer(
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/diff_gaussian_rasterization/init.py", line 213, in forward
return rasterize_gaussians(
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/diff_gaussian_rasterization/init.py", line 32, in rasterize_gaussians
return _RasterizeGaussians.apply(
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/torch/autograd/function.py", line 598, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/diff_gaussian_rasterization/init.py", line 92, in forward
num_rendered, color, depth, alpha, radii, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)
RuntimeError: means3D must have dimensions (num_points, 3)
I successfully launch the exp, but I get this errors, what's the issues
I run this command:
python app_stage1.py big --resume path/to/LGM/model_fp16.safetensors --condition_type $condition_type
and I put one depth image with prompt "a penguin with husky dog costume",
Then I get:
text_inputs = self.tokenizer(
prompt,
padding=True,
max_length=self.tokenizer.model_max_length,
truncation=True,
return_tensors="pt",
)
Traceback (most recent call last):
File "/public/omniteam/yejr/conda_env/MVcontrol/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 717, in convert_to_tensors
tensor = as_tensor(value)
RuntimeError: Could not infer dtype of NoneType
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "", line 1, in
File "/public/omniteam/yejr/conda_env/MVcontrol/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2538, in call
encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
File "/public/omniteam/yejr/conda_env/MVcontrol/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2644, in _call_one
return self.encode_plus(
File "/public/omniteam/yejr/conda_env/MVcontrol/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2717, in encode_plus
return self._encode_plus(
File "/public/omniteam/yejr/conda_env/MVcontrol/lib/python3.10/site-packages/transformers/tokenization_utils.py", line 652, in _encode_plus
return self.prepare_for_model(
File "/public/omniteam/yejr/conda_env/MVcontrol/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3207, in prepare_for_model
batch_outputs = BatchEncoding(
File "/public/omniteam/yejr/conda_env/MVcontrol/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 210, in init
self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
File "/public/omniteam/yejr/conda_env/MVcontrol/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 733, in convert_to_tensors
raise ValueError(
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (input_ids
in this case) have excessive nesting (inputs type list
where type int
is expected).
Could you please tell me how to solve this?
I'm encountering multiple issues when running the app_stage1.py script with the provided model. Here is a summary of the bugs:
While trying to load the lzq49/mvdream-sd21-diffusers model pipeline using MVControlPipeline.from_pretrained, I receive the following error:
bash
Copy code
ValueError:
The repository for lzq49/mvdream-sd21-diffusers contains custom code in unet/unet.py, camera_proj/camera_proj which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/lzq49/mvdream-sd21-diffusers/unet/unet.py, https://hf.co/lzq49/mvdream-sd21-diffusers/camera_proj/camera_proj.py.
Please pass the argument trust_remote_code=True
to allow custom code to be run.
Suggested Fix:
Adding the trust_remote_code=True argument to the pipeline loading functions resolves the issue. Here is an example modification in app_stage1.py:
python
Copy code
from pipeline_mvcontrol import load_mvcontrol_pipeline
pipe_mvcontrol = load_mvcontrol_pipeline(
"lzq49/mvdream-sd21-diffusers",
revision="main",
torch_dtype=torch.float16,
trust_remote_code=True # Add this argument here
)
Or, directly with MVControlPipeline.from_pretrained:
python
Copy code
from diffusers import MVControlPipeline
pipe_mvcontrol = MVControlPipeline.from_pretrained(
"lzq49/mvdream-sd21-diffusers",
torch_dtype=torch.float16,
trust_remote_code=True # Add this argument here
)
2. Deprecation Warnings:
During execution, the following warnings are displayed:
vbnet
Copy code
/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
torch.utils._pytree._register_pytree_node(
/home/dubaiprince/miniconda3/envs/mvcontrol/lib/python3.9/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: resume_download
is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True
.
warnings.warn(
Suggested Fixes:
Update the usage of _register_pytree_node in diffusers/utils/outputs.py:
python
Copy code
torch.utils._pytree.register_pytree_node( # Replace _register_pytree_node
Update the resume_download usage in huggingface_hub/file_download.py:
python
Copy code
force_download=True # Instead of resume_download
3. xFormers Warning:
A warning related to xFormers is also raised:
arduino
Copy code
/home/dubaiprince/Projects/MVControl-threestudio/extern/lgm/attention.py:22: UserWarning: xFormers is available (Attention)
warnings.warn("xFormers is available (Attention)")
Suggested Fix:
Ensure that xFormers is compatible with the current environment by aligning the versions of Python, PyTorch, and CUDA.
Reproduction Steps:
Run the command python app_stage1.py big --resume pretrained/model_fp16.safetensors --condition_type $condition_type
Observe the error and warnings.
Environment:
OS: Ubuntu 20.04
Python Version: 3.9.19
Torch Version: 2.3.0
Torchvision Version: 0.17.1+cu118
Let me know if you need more details, or if I can assist with testing any fixes.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.