This is the TensorRT optimized Wav2Lip pipeline, thanks to Rudrabha for the wonderful Wav2Lip implementation at here, https://github.com/Rudrabha/Wav2Lip/tree/master. Please refer to the original repo for tech background.
Please refer the original repo for system config, here is my config scripts just for reference, I modified requirement.txt to use latest torch.
#follow the instructions here for system config, https://github.com/Rudrabha/Wav2Lip/tree/master.
#download sample video and audio here, https://bhaasha.iiit.ac.in/lipsync/.
#download pre-trained models from here, https://github.com/Rudrabha/Wav2Lip/tree/master.
clear && CUDA_VISIBLE_DEVICES=1 /usr/local/bin/nsys profile /opt/conda/bin/python inference.py --checkpoint_path ./checkpoints/wav2lip.pth --face ./samples/game.mp4 --audio ./samples/game.wav --export_wav2lip_onnx=./onnx_trt/wav2lip_bs1_fp32_6000.onnx --export_wav2lip_trt=./onnx_trt/wav2lip bs1_fp32_6000.engine --warm_up=2 --export s3fd _onnx=./onnx_trt/s3fd_bs43_fp32_6000.onnx --export_s3fd _trt=./onnx_trt/s3fd_bs43_fp32_6000.engine
I use system config:
Ubuntu 18.04
GPU Driver 530
CUDA 11.7
TensorRT 8.6.0
GPU RTX 6000
Torch 2.0.1+cu117
The main performance speed up comes from torch native GPU AI inference converted to TensorRT counterpart, with same float32 precision, and s3fd AI inference overlapping with its post-processing.
Please check the optimizing document for details.
The performance speed up for inference part (s3fd+wav2lip) is 4.1X: