Emotion-GCN: Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition
This repository hosts the official PyTorch implementation of our paper "Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition" accepted at IEEE FG 2021.
Paper: https://arxiv.org/abs/2106.03487
Authored by: Panagiotis Antoniadis, Panagiotis Paraskevas Filntisis, Petros Maragos
Over the past few years, deep learning methods have shown remarkable results in many face-related tasks including automatic facial expression recognition (FER) in-the-wild. Meanwhile, numerous models describing the human emotional states have been proposed by the psychology community. However, we have no clear evidence as to which representation is more appropriate and the majority of FER systems use either the categorical or the dimensional model of affect. Inspired by recent work in multi-label classification, this paper proposes a novel multi-task learning (MTL) framework that exploits the dependencies between these two models using a Graph Convolutional Network (GCN) to recognize facial expressions in-the-wild. Specifically, a shared feature representation is learned for both discrete and continuous recognition in a MTL setting. Moreover, the facial expression classifiers and the valence-arousal regressors are learned through a GCN that explicitly captures the dependencies between them. To evaluate the performance of our method under real-world conditions we train our models on AffectNet dataset. The results of our experiments show that our method outperforms the current state-of-the-art methods on discrete FER.
- Download the dataset. [AffectNet] [Aff-Wild2]
- Download the 300-dimensional GloVe vectors trained on the Wikipedia dataset from here.
- Run
pickle_annotations_affectnet.pyandpickle_annotations_affwild2.pyto generatedata_affectnet.pklanddata_affwild2.pkl.
- Train Emotion-GCN on a FER dataset:
python main.py --help
usage: main.py [-h] [--image_dir IMAGE_DIR] [--data DATA] [--dataset {affectnet,affwild2}] [--network {densenet,bregnext}] [--adj ADJ]
[--emb EMB] [--workers WORKERS] [--batch_size BATCH_SIZE] [--model {single_task,multi_task,emotion_gcn}] [--epochs EPOCHS]
[--lambda_multi LAMBDA_MULTI] [--lr LR] [--momentum MOMENTUM] --gpu GPU --saved_model SAVED_MODEL
Train Facial Expression Recognition model using Emotion-GCN
optional arguments:
-h, --help show this help message and exit
--image_dir IMAGE_DIR
path to images of the dataset
--data DATA path to the pickle file that holds all the information for each sample
--dataset {affectnet,affwild2}
Dataset to use (default: affectnet)
--network {densenet,bregnext}
Network to use (default: densenet)
--adj ADJ path to the pickle file that holds the adjacency matrix
--emb EMB path to the pickle file that holds the word embeddings
--workers WORKERS number of data loading workers (default: 4)
--batch_size BATCH_SIZE
size of each batch (default: 35)
--model {single_task,multi_task,emotion_gcn}
Model to use (default: emotion_gcn)
--epochs EPOCHS number of total epochs to train the network (default: 10)
--lambda_multi LAMBDA_MULTI
lambda parameter of loss function
--lr LR learning rate (default: 0.001)
--momentum MOMENTUM momentum parameter of SGD (default: 0.9)
--gpu GPU id of gpu device to use
--saved_model SAVED_MODEL
name of the saved model
We also provide weights for our Emotion-GCN models on AffectNet and Aff-Wild2. Our best model achieves 66.46% accuracy on the categorical model of AffectNet outperforming the performance of the current state-of-the-art methods for discrete FER. You can download the pre-trained models here.
@inproceedings{9667014,
author={Antoniadis, Panagiotis and Filntisis, Panagiotis Paraskevas and Maragos, Petros},
booktitle={2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)},
title={Exploiting Emotional Dependencies with Graph Convolutional Networks for Facial Expression Recognition},
year={2021},
pages={1-8},
doi={10.1109/FG52635.2021.9667014}}
For questions feel free to open an issue.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent