tencentarc / motionctrl Goto Github PK
View Code? Open in Web Editor NEWOfficial Code for MotionCtrl [SIGGRAPH 2024]
Home Page: https://wzhouxiff.github.io/projects/MotionCtrl/
License: Apache License 2.0
Official Code for MotionCtrl [SIGGRAPH 2024]
Home Page: https://wzhouxiff.github.io/projects/MotionCtrl/
License: Apache License 2.0
When I use the command sh configs/inference/run.sh
, this error occurs:
Traceback (most recent call last):
File "/data/code/MotionCtrl/main/evaluation/motionctrl_inference.py", line 354, in <module>
run_inference(args, gpu_num, rank)
File "/data/code/MotionCtrl/main/evaluation/motionctrl_inference.py", line 261, in run_inference
batch_samples = motionctrl_sample(
File "/data/code/MotionCtrl/main/evaluation/motionctrl_inference.py", line 146, in motionctrl_sample
cond = model.get_learned_conditioning(prompts)
File "/data/code/MotionCtrl/main/evaluation/../../lvdm/models/ddpm3d.py", line 552, in get_learned_conditioning
c = self.cond_stage_model.encode(c)
File "/data/code/MotionCtrl/main/evaluation/../../lvdm/modules/encoders/condition2.py", line 238, in encode
return self(text)
File "/root/miniconda3/envs/motionctrl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/data/code/MotionCtrl/main/evaluation/../../lvdm/modules/encoders/condition2.py", line 215, in forward
z = self.encode_with_transformer(tokens.to(self.device))
File "/data/code/MotionCtrl/main/evaluation/../../lvdm/modules/encoders/condition2.py", line 222, in encode_with_transformer
x = self.text_transformer_forward(x, attn_mask=self.model.attn_mask)
File "/data/code/MotionCtrl/main/evaluation/../../lvdm/modules/encoders/condition2.py", line 234, in text_transformer_forward
x = r(x, attn_mask=attn_mask)
File "/root/miniconda3/envs/motionctrl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/motionctrl/lib/python3.10/site-packages/open_clip/transformer.py", line 263, in forward
x = q_x + self.ls_1(self.attention(q_x=self.ln_1(q_x), k_x=k_x, v_x=v_x, attn_mask=attn_mask))
File "/root/miniconda3/envs/motionctrl/lib/python3.10/site-packages/open_clip/transformer.py", line 250, in attention
return self.attn(
File "/root/miniconda3/envs/motionctrl/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/root/miniconda3/envs/motionctrl/lib/python3.10/site-packages/torch/nn/modules/activation.py", line 1167, in forward
attn_output, attn_output_weights = F.multi_head_attention_forward(
File "/root/miniconda3/envs/motionctrl/lib/python3.10/site-packages/torch/nn/functional.py", line 5069, in multi_head_attention_forward
raise RuntimeError(f"The shape of the 2D attn_mask is {attn_mask.shape}, but should be {correct_2d_size}.")
RuntimeError: The shape of the 2D attn_mask is torch.Size([77, 77]), but should be (1, 1).
May I know what is the license for the motionctrl_svd.ckpt?
Thanks a lot for your excellent work!
Do you have the plan to released the trained model?
Hi, thanks for this incredible work you've done. I'm wondering how many cuda vram needed to train motionctrl based on svd? Thank you!
is there any scripts or tutorial about that
First of all, great work TencentARC team! We love it!
I know that this is not the repo on ComfyUI node. https://github.com/chaojie/ComfyUI-MotionCtrl but since there I can't put in a Issue I have to ask here.
I download the example workflow and I get this error. Any ideas?
Motionctrl Sample Simple:
Hi @wzhouxiff ,
thanks for the interesting work!
I downloaded the model here https://huggingface.co/TencentARC/MotionCtrl/blob/main/motionctrl_svd.ckpt for SVD+MotionCtrl inference. All keys add up, but the results are basically noises. Do you maybe have an idea what causes this? Thanks a lot!
Hi, thanks for sharing the great work! I have a little question about the camera control.
I wonder does the camera speed value have its practical meaning? say, when setting speed=0.5, what does the value 0.5 refer to? or is it just a relative indicator? Also, could you please give me some hint how to do if I want to move the camera for a certain distance, like pan right for 20 pixels? Thanks for your help :)
Is there any way to increase the video seconds? Is it currently only possible to generate 1 second videos?
Hello, i want to animate clouds on static shot with MotionCtrl on SVD. What parameters to use to achive my goal?
I tried [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
but it generate random motion, not static camera.
From my understanding, CMCM digests over a set of poses represented by extrinsic matrix. But then how does it deal with scale ambiguity issue on SVD case?
您好,请问在readme中展示的比较大尺度的运动的json文件有在dataset中吗?
具体是哪几个?谢谢~
Hi, I found that when using the model to generate video, there is always a very serious Rolling Shutter Distortion causing the lines to bend at each frame, is there any way to solve this problem? Thx
when will this great project release code?
@yxgeee @xinntao @wzhouxiff @eltociear @yeliudev
Hello, Thank you for releasing great work.
I have questions about SVD Model Inference Code.
Why do you multiply np.array([3, 1, 4]) to pose?? Can you tell me it in details?
And if I want to train the model with lower resolution(1024x576 -> 512x320), how do i modify the code?
Thank you in advance
MotionCtrl+svd是在svd模型上训练的CMCM吗
Hi, I tested the demo on huggingface. My settings are: 1. control both camera and object motion; 2. the camera poses and trajectory: motion: ['Zoom Out']. speed: 1.0.; 3. trajectories is horizon_1. the prompt is: “A panda crawling on rocks”. Other parameters are default. But I got the following video, It seems that the control is not very effective.
Any plan to release the training code?
Hey,
Upon examining the weights of motionctrl_svd checkpoint, found that there were no weights for omcm module.
Are you planning to release them?
Thanks!
Traceback (most recent call last):
File "D:\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1800, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 940, in exec_module
File "", line 241, in call_with_frames_removed
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-MotionCtrl_init.py", line 1, in
from .nodes import NODE_CLASS_MAPPINGS
File "D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-MotionCtrl\nodes.py", line 21, in
from pytorch_lightning import seed_everything
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning_init_.py", line 35, in
from pytorch_lightning.callbacks import Callback # noqa: E402
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\callbacks_init_.py", line 14, in
from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\callbacks\batch_size_finder.py", line 24, in
from pytorch_lightning.callbacks.callback import Callback
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\callbacks\callback.py", line 25, in
from pytorch_lightning.utilities.types import STEP_OUTPUT
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\pytorch_lightning\utilities\types.py", line 27, in
from torchmetrics import Metric
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics_init_.py", line 22, in
from torchmetrics import functional # noqa: E402
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\functional_init_.py", line 14, in
from torchmetrics.functional.audio.deprecated import permutation_invariant_training as permutation_invariant_training
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\functional\audio_init.py", line 14, in
from torchmetrics.functional.audio.pit import permutation_invariant_training, pit_permutate
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\functional\audio\pit.py", line 22, in
from torchmetrics.utilities import rank_zero_warn
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities_init.py", line 14, in
from torchmetrics.utilities.checks import check_forward_full_state_property
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities\checks.py", line 25, in
from torchmetrics.metric import Metric
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\metric.py", line 30, in
from torchmetrics.utilities.data import (
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities\data.py", line 22, in
from torchmetrics.utilities.imports import TORCH_GREATER_EQUAL_1_12, XLA_AVAILABLE
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchmetrics\utilities\imports.py", line 54, in
TORCHAUDIO_GREATER_EQUAL_0_10: Optional[bool] = compare_version("torchaudio", operator.ge, "0.10.0")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\lightning_utilities\core\imports.py", line 77, in compare_version
pkg = importlib.import_module(package)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "importlib_init.py", line 126, in import_module
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchaudio_init.py", line 1, in
from torchaudio import ( # noqa: F401
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchaudio_extension_init.py", line 43, in
_load_lib("libtorchaudio")
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torchaudio_extension\utils.py", line 61, in load_lib
torch.ops.load_library(path)
File "D:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_ops.py", line 852, in load_library
ctypes.CDLL(path)
File "ctypes_init.py", line 376, in init
OSError: [WinError 127] 找不到指定的程序。
pls advise how to solve this. thx
It appears that the SVD version supports only camera motion without the object trajectory. Can you provide some insight for omitting it?
Looking forward to all the updates to this app
我使用的是cuda121,torch2.2.1,xformers0.0.25,然后运行gradio之后会提示:
AttributeError: module 'xformers' has no attribute 'ops'. Did you mean: 'os'?
尝试过把torch+xformers一系列都降级,但是提示CUDA与这些版本不匹配,有没有办法不降级CUDA的情况下解决这个问题呢
(其他环境需要比较新的CUDA
具体报错如下:
1
[0. 0. 1.] [0. 0. 0.]
1
[0. 0. 1.] [0. 0. 0.]
C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\functional.py:5476: UserWarning:
1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:263.)
Traceback (most recent call last):
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\gradio\routes.py", line 439, in run_predict
output = await app.get_blocks().process_api(
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\gradio\blocks.py", line 1389, in process_api
result = await self.call_function(
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\gradio\blocks.py", line 1094, in call_function
prediction = await anyio.to_thread.run_sync(
File "C:\Users\admin\AppData\Roaming\Python\Python310\site-packages\anyio\to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "C:\Users\admin\AppData\Roaming\Python\Python310\site-packages\anyio_backends_asyncio.py", line 2144, in run_sync_in_worker_thread
return await future
File "C:\Users\admin\AppData\Roaming\Python\Python310\site-packages\anyio_backends_asyncio.py", line 851, in run
result = context.run(func, *args)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\gradio\utils.py", line 704, in wrapper
response = f(*args, **kwargs)
File "D:\AIGC\MotionCtrl-svd\app.py", line 514, in model_run
video_path = motionctrl_sample(
File "D:\AIGC\MotionCtrl-svd\gradio_utils\motionctrl_cmcm_gradio.py", line 148, in motionctrl_sample
c, uc = model.conditioner.get_unconditional_conditioning(
File "D:\AIGC\MotionCtrl-svd\sgm\modules\encoders\modules.py", line 179, in get_unconditional_conditioning
c = self(batch_c, force_cond_zero_embeddings)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "D:\AIGC\MotionCtrl-svd\sgm\modules\encoders\modules.py", line 132, in forward
emb_out = embedder(batch[embedder.input_key])
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "D:\AIGC\MotionCtrl-svd\sgm\modules\encoders\modules.py", line 1012, in forward
out = self.encoder.encode(vid[n * n_samples : (n + 1) * n_samples])
File "D:\AIGC\MotionCtrl-svd\sgm\models\autoencoder.py", line 472, in encode
z = self.encoder(x)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "D:\AIGC\MotionCtrl-svd\sgm\modules\diffusionmodules\model.py", line 596, in forward
h = self.mid.attn_1(h)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
return self.call_impl(*args, **kwargs)
File "C:\Users\admin\miniconda3\envs\motionctrl\lib\site-packages\torch\nn\modules\module.py", line 1520, in call_impl
return forward_call(*args, **kwargs)
File "D:\AIGC\MotionCtrl-svd\sgm\modules\diffusionmodules\model.py", line 265, in forward
h = self.attention(h)
File "D:\AIGC\MotionCtrl-svd\sgm\modules\diffusionmodules\model.py", line 250, in attention
out = xformers.ops.memory_efficient_attention(
AttributeError: module 'xformers' has no attribute 'ops'. Did you mean: 'os'?
Thank you for providing the great work.
Do you have a plan to release the training code and dataset?
Thank you in advance
@wzhouxiff Thanks for the interesting work!
When I reproduce the SVD version,the results are basically noises.
Problem consistency #13
the Demo of huggingface is a Erro
There is another problem. The prompt resolution of the gradient interface is 576x1024, but the background prompt model input resolution is 512x1024,What is the correct input resolution?
Finally, we hope to have a Dockerfile file or image to enable SVD to run completely.
in demo script configs/inference/run.sh, CUDA_VISIBLE_DEVICES=7 was used causing "No CUDA GPUs are available" for those who doesn't have 8 or more gpus, please consider to remove it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.