The code for the paper "Generating Talking Face Landmarks from Speech."

You can find the project page here.

An improved version of this project can be found here.

• Keras --- 2.2.4
• Tensorflow --- 1.9.0
• Librosa --- 0.6.0
• opencv-python --- 3.3.0.10
• dlib --- 19.7.0
• tqdm
• subprocess

• ffmpeg --- 3.4.1
• OpenCV --- 3.3.0

The code has been tested on Ubuntu 16.04 and OS X Sierra and High Sierra.

The generation code has the following arguments:

• -i --- Input speech file
  • See this link for supported audio formats.
• -m --- Input talking face landmarks model
• -d --- Delay in terms of frames, where one frame is 40 ms
• -c --- Number of context frames
• -o --- Output path

You can run the following code to test the system:

python generate.py -i test_samples/test1.flac -m models/D40_C3.h5 -d 1 -c 3 -o results/D40_C3_test1

You can run featureExtractor.py to extract features from videos directly. The arguments are as follows:

• -vp --- Input folder containing video files (if your video file types are different from .mpg or .mp4, please modify the script accordingly)
• -sp --- Path to shape_predictor_68_face_landmarks.dat. You can download this file here.
• -o --- Output file name

Usage:

python featureExtractor.py -vp path-to-video-files/ -sp path-to-shape-predictor-68-face-landmarks-dat -o output-file-name.hdf5

The training code has the following arguments:

• -i --- Input hdf5 file containing training data
• -u --- Number of hidden units
• -d --- Delay in terms of frames, where one frame is 40 ms
• -c --- Number of context frames
• -o --- Output folder path to save the model

Usage:

python train.py -i path-to-hdf5-train-file/ -u number-of-hidden-units -d number-of-delay-frames -c number-of-context-frames -o output-folder-to-save-model-file

@inproceedings{eskimez2018generating,
  title={Generating talking face landmarks from speech},
  author={Eskimez, Sefik Emre and Maddox, Ross K and Xu, Chenliang and Duan, Zhiyao},
  booktitle={International Conference on Latent Variable Analysis and Signal Separation},
  pages={372--381},
  year={2018},
  organization={Springer}
}