๐ MotionCtrl for [SVD], for [VideoCrafter]
motionctrl_small.mp4
๐ฅ๐ฅ We release the codes, models and demos for MotionCtrl on Stable Video Diffusion (SVD).
Official implementation of MotionCtrl: A Unified and Flexible Motion Controller for Video Generation.
MotionCtrl can Independently control complex camera motion and object motion of generated videos, with only a unified model.
More results are in showcase_svd and our Project Page.
More results are in our Project Page.
- 20231225: Release MotionCtrl deployed on LVDM/VideoCrafter.
- 20231225: Gradio demo available. ๐ค
- 20231228: Provide local gradio demo for convenience.
- 20240115 More camera poses used for testing are provided in
dataset/camera_poses
- 20240115 Release MotionCtrl deployed on SVD. Codes are in the brach svd and Gradio Demo is available in ๐ค
.
- Release MotionCtrl deployed on AnimateDiff.
conda create -n motionctrl python=3.10.6
conda activate motionctrl
pip install -r requirements.txt
- Download the weights of MotionCtrl motionctrl.pth and put it to
./checkpoints
. - Go into
configs/inference/run.sh
and setcondtype
as 'camera_motion', 'object_motion', or 'both'.
condtype=camera_motion
means only control the camera motion in the generated video.condtype=object_motion
means only control the object motion in the generated video.condtype=both
means control the camera motion and object motion in the generated video simultaneously.
- Running scripts: sh configs/inference/run.sh
-
python -m app --share
If you make use of our work, please cite our paper.
@inproceedings{wang2023motionctrl,
title={MotionCtrl: A Unified and Flexible Motion Controller for Video Generation},
author={Wang, Zhouxia and Yuan, Ziyang and Wang, Xintao and Chen, Tianshui and Xia, Menghan and Luo, Ping and Shan, Yin},
booktitle={arXiv preprint arXiv:2312.03641},
year={2023}
}
The current version of MotionCtrl is built on VideoCrafter. We appreciate the authors for sharing their awesome codebase.
For any question, feel free to email [email protected]
or [email protected]
.