snowzhangy / atvgnet Goto Github PK

By Lele Chen , Ross K Maddox, Zhiyao Duan, Chenliang Xu.

University of Rochester.

0. Introduction
1. Citation
2. Running
3. Model
4. Results
5. Disclaimer and known issues

This repository contains the original models (AT-net, VG-net) described in the paper Hierarchical Cross-modal Talking Face Generation with Dynamic Pixel-wise Loss. This code can be applied directly in LRW and GRID. The outputs from the model are visualized here: the first one is the synthesized landmark from ATnet, the rest of them are attention, motion map and final results from VGnet.

If you use any codes, models or the ideas from this repo in your research, please cite:

@InProceedings{Chen_2019_CVPR,
author = {Lele Chen, Ross K Maddox, Zhiyao Duan, Chenliang Xu},
title = {Hierarchical Cross-modal Talking Face Generation with Dynamic Pixel-wise Loss},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2019}
}

0. This code is tested under Python 2.7.

1. Pytorch environment:Pytorch 0.4.1. (conda install pytorch=0.4.1 torchvision cuda90 -c pytorch)

2. Install requirements.txt (pip install -r requirement.txt)

3. Download the pretrained ATnet and VGnet weights at google drive. Put the weights under model folder.

4. Run the demo code: python demo.py

   • -device_ids: gpu id
   • -cuda: using cuda or not
   • -vg_model: pretrained VGnet weight
   • -at_model: pretrained ATnet weight
   • -lstm: use lstm or not
   • -p: input example image
   • -i: input audio file
   • -lstm: use lstm or not
   • -sample_dir: folder to save the outputs
   • ...

5. Download and unzip the training data from LRW

6. Preprocess the data (Extract landmark and crop the image by dlib).

7. Train the ATnet model: python aetnet.py

   • -device_ids: gpu id
   • -batch_size: batch size
   • -model_dir: folder to save weights
   • -lstm: use lstm or not
   • -sample_dir: folder to save visualized images during training
   • ...

8. Test the model: python atnet_test.py

   • -device_ids: gpu id
   • -batch_size: batch size
   • -model_name: pretrained weights
   • -sample_dir: folder to save the outputs
   • -lstm: use lstm or not
   • ...

9. Train the VGnet: python vgnet.py

   • -device_ids: gpu id
   • -batch_size: batch size
   • -model_dir: folder to save weights
   • -sample_dir: folder to save visualized images during training
   • ...

10. Test the VGnet: python vgnet_test.py

    • -device_ids: gpu id
    • -batch_size: batch size
    • -model_name: pretrained weights
    • -sample_dir: folder to save the outputs
    • ...

0. Overall ATVGnet

1. Regresssion based discriminator network

   

0. Result visualization on different datasets:

   

1. Reuslt compared with other SOTA methods:

   

2. The studies on image robustness respective with landmark accuracy:

   

3. Quantitative results:

   

0. These codes are implmented in Pytorch
1. In this paper, we train LRW and GRID seperately.
2. The model are sensitive to input images. Please use the correct preprocessing code.
3. I didn't finish the data processing code yet. I will release it soon. But you can try the model and replace with your own image.
4. If you want to train these models using this version of pytorch without modifications, please notice that:
   • You need at lest 12 GB GPU memory.
   • There might be some other untested issues.
5. There is another intresting and useful research on audio to landmark genration. Please check it out at https://github.com/eeskimez/Talking-Face-Landmarks-from-Speech.

• Release training data

MIT