aue-crl's Introduction

AU-Expression Knowledge Constrained Representation Learning for Facial Expression Recognition

Implementation of paper:

AU-Expression Knowledge Constrained Representation Learning for Facial Expression Recognition
IEEE International Conference on Robotics and Automation (ICRA), 2021.
Tao Pu, Tianshui Chen, Yuan Xie, Hefeng Wu, and Liang Lin.

Environment

Ubuntu 16.04 LTS, Python 3.5, PyTorch 1.3

Usage

# Step 1: Train the branch of facial expression recognition
python main.py --Model ResNet-101 --Experiment EM
# Step 2: Train the branch of facial AU recognition
python main.py --Model ResNet-101 --Experiment AU --Resume_Model <yourCheckpointPath>
# Step 3: Train whole model
python main.py --Model ResNet-101 --Experiment Fuse --Resume_Model <yourCheckpointPath>

Note: At step 2 and 3, you should load the checkpoint from the previous step.

Result

Result on RAF-DB

Methods	Angry	Disgust	Fear	Happy	Neutral	Sad	Surprised	Ave. acc
DCNN-DA	78.4	64.4	62.2	91.1	80.6	81.2	84.5	77.5
WSLGRN	75.3	56.9	63.5	93.8	85.4	83.5	85.4	77.7
CP	80.0	61.0	61.0	93.0	89.0	86.0	86.0	79.4
CompactDLM	74.5	67.6	46.9	82.3	59.1	58.0	84.6	67.6
FSN	72.8	46.9	56.8	90.5	76.9	81.6	81.8	72.5
DLP-CNN	71.6	52.2	62.2	92.8	80.3	80.1	81.2	74.2
MRE-CNN	84.0	57.5	60.8	88.8	80.2	79.9	86.0	76.7
Ours	80.5	67.6	68.9	94.1	85.8	83.6	86.4	81.0

Result on SFEW2.0

Methods	Angry	Disgust	Fear	Happy	Neutral	Sad	Surprised	Ave. acc
CP	66.0	0.0	14.0	90.0	86.0	66.0	29.0	50.1
DLP-CNN	-	-	-	-	-	-	-	51.1
IA-CNN	70.7	0.0	8.9	70.4	60.3	58.8	28.9	42.6
IL	61.0	0.0	6.4	89.0	66.2	48.0	33.3	43.4
Ours	75.3	17.4	25.5	86.3	72.1	50.7	42.1	52.8

Citation

@inproceedings{Pu2021AUE-CRL,
  author={Pu, Tao and Chen, Tianshui and Xie, Yuan and Wu, Hefeng and Lin, Liang},
  title={Au-expression knowledge constrained representation learning for facial expression recognition},
  booktitle={2021 IEEE international conference on robotics and automation (ICRA)},
  year={2021},
  pages={11154--11161},
  publisher={IEEE},
  doi={10.1109/ICRA48506.2021.9561252}
}

Contributors

For any questions, feel free to open an issue or contact us:

aue-crl's People

Contributors

Stargazers

Watchers

aue-crl's Issues

can you offer RAF Landmarks_68_FAN_aligned file?

I have used MTCNN to get facial landmarks, but some images can not detect face? so, can you offer complete landmarks file of RAF? Thank you so much！

Pre-trained models

Thanks for your work~
Have you released the pre-trained models for step1 and step2? Where can I get it? It will be much convenient for us to start from step 3 .

关于Landmarks_68_FAN_bbox和id_train_list.txt文件

您好，在utils文件中，您读取了Landmarks_68_FAN_bbox、id_train_list.txt、label.lst文件，请问这几个文件您是怎样得到的，我应该怎样获得？

LandmarkPath = dataPath_prefix + '/RAF/basic/Annotation/Landmarks_68_FAN_bbox/'
Train_List = dataPath_prefix + '/SFEW/list_experiment/id_train_list.txt'
Train_List = dataPath_prefix + '/ExpW/data/label/label.lst'

Recommend Projects