• Combining AnimateDiff with Multi-ControlNet and Img2Img for Vid2Vid applications. This small library is especially focused on Vid2Vid applications by using ControlNet (or Multi-ControlNet) to guide the video generation and AnimateDiff for concistency.
• In addition it uses Img2Img for creating more consistent videos (after the first epoch). Similar to AnimateDiff it allows the use of DreamBooth/LoRA models in addition to the Stable Diffusion 1.5 base model.
• This is an initial release so please expect potential issues and bugs. Feedback, suggestions, and feature requests are welcome.

• 🔥 Nov. 7, 2023 - Now supporting Latent Consistency Model (LCM) - Achieving 10X performance gain!

• This codebase was tested on Linux (Ubuntu 22.04) only. It was tested on an Intel machine with NVIDIA Gefore RTX 3090 (24 GB VRAM) and requires at least 16 GB of RAM.

• Make sure you have Anaconda installed (https://www.anaconda.com/download).
• Also make sure that FFMPEG is properly installed and set up (you can follow these guides for the installation: "Guide 1" and if there are still issues this: "Guide 2" - You can set the FFMPEG path in the configs/prompts yaml files)

git clone [email protected]:intellerce/controlanimate.git
cd ControlAnimate

bash download.sh

conda env create -f env.yml

• After setting the config file 'configs/prompts/SampleConfig.yaml', simply run the following (don't forget to point to a valid input video file):

conda activate controlanimate
bash start.sh

Tested on a machine with a single RTX 3090.

• Prompt weighting is based on Compel. You can use + or (...)+ for importance or add weights like this: (cat)1.2 Similarly you can use the negative sign (-) to reduce the weight or use weights below 1. Please refer to https://github.com/damian0815/compel/blob/main/Reference.md for more info.

• Four ControlNets and Latent Overlapping
• LCM (No ControlNet)

This is an initial release so please expect some potential bugs and issues. Currently, memory optimizations using xformers does not work properly (for the Motion Module only) and leads to some unclear errors. So it has been partially disabled in this release (for the Motion Modules). The code was tested on Linux and will not work on Windows currently (at least the FFMPEGProcessor needs to be modified).

• GitHub Release
• Bug Fixes and Improvements
• Fixing xformers Issues and GPU Memory Optimization
• Windows Support
• Interface

Hamed Omidvar, Ph.D.: [email protected]
Vahideh Akhlaghi, Ph.D.: [email protected]

This codebase is released under the Apache v2.0 license. For the licenses of the codebases that this repository is based on please refer to their corresponding Github/Website pages.

This codebase was built upon and/or inspired by the following repositories: AnimateDiff Diffusers Video2Video

The authors would like to thank Kalin Ovtcharov (Extropolis Corp.) for invaluable feedback and suggestions.