tencentarc / motionctrl Goto Github PK

🔥🔥 We release the codes, models and demos for MotionCtrl on Stable Video Diffusion (SVD).

Official implementation of MotionCtrl: A Unified and Flexible Motion Controller for Video Generation.

MotionCtrl can Independently control complex camera motion and object motion of generated videos, with only a unified model.

More results are in showcase_svd and our Project Page.

More results are in our Project Page.

• 20231225: Release MotionCtrl deployed on LVDM/VideoCrafter.
• 20231225: Gradio demo available. 🤗
• 20231228: Provide local gradio demo for convenience.
• 20240115 More camera poses used for testing are provided in dataset/camera_poses
• 20240115 Release MotionCtrl deployed on SVD. Codes are in the branch svd and Gradio Demo is available in 🤗 .
• ❗❗❗ Gradio demo of MotionCtrl deployed on VideoCrafter2 is available in 🤗 . You can also run it locally by python -m app --share.

• ❗❗❗ Release MotionCtrl deployed on AnimateDiff are available in branch animatediff, containing both training and inference code.

conda create -n motionctrl python=3.10.6
conda activate motionctrl
pip install -r requirements.txt

• Run local inference script

1. Download the weights of MotionCtrl motionctrl.pth and put it to ./checkpoints.
2. Go into configs/inference/run.sh and set condtype as 'camera_motion', 'object_motion', or 'both'.

• condtype=camera_motion means only control the camera motion in the generated video.
• condtype=object_motion means only control the object motion in the generated video.
• condtype=both means control the camera motion and object motion in the generated video simultaneously.

3. Running scripts: sh configs/inference/run.sh

• Run local gradio demo

  python -m app --share
  

• RealEstate10K

  1. Following https://github.com/cashiwamochi/RealEstate10K_Downloader to download and process the videos.
  2. Corresponding Captions and List are provided in GoogleDrive.

If you make use of our work, please cite our paper.

@inproceedings{wang2024motionctrl,
  title={Motionctrl: A unified and flexible motion controller for video generation},
  author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Li, Yaowei and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Ying},
  booktitle={ACM SIGGRAPH 2024 Conference Papers},
  pages={1--11},
  year={2024}
}

The current version of MotionCtrl is built on VideoCrafter. We appreciate the authors for sharing their awesome codebase.

For any question, feel free to email [email protected] or [email protected].