Affective Expression Analysis in-the-wild using Multi-Task Temporal Statistical Deep Learning Model

Challenges: The First Affective Behavior Analysis in-the-wild (ABAW) Competition
Homepage: https://ibug.doc.ic.ac.uk/resources/fg-2020-competition-affective-behavior-analysis/
Team Name: CNU_ADL
Team Members:
(1) Nhu-Tai Do, donhutai@gmail.com
(2) Tram-Tran Nguyen Quynh, tramtran2@gmail.com
(3) Soo-Hyung Kim
Affiliation: Chonnam National University, South Korea

Our paper:

Affective Expression Analysis in-the-wild using Multi-Task TemporalStatistical Deep Learning Model
Link: https://arxiv.org/abs/2002.09120

@article{Do2020,
	archivePrefix = {arXiv},
	arxivId = {2002.09120},
	author = {Do, Nhu-Tai and Kim, Soo-Hyung},
	eprint = {2002.09120},
	month = {Feb},
	title = {{Affective Expression Analysis in-the-wild using Multi-Task Temporal Statistical Deep Learning Model}},
	url = {http://arxiv.org/abs/2002.09120},
	year = {2020}
}

How to run

1. Download and setup Anaconda3
2. Run setup_envs.sh to install conda environments with Python 3.7, keras, tensorflow, etc.
   
3. Unzip two pandas index files of Aff-Wild2 dataset: affwild2_cropped_aligned_frames_v1.zip and affwild2_cropped_frames_v1.zip in [data/AffWild2/data] folder
4. Download and setup Aff-Wild2 dataset:
   1. Extract annotations.zip and copy 3 folder AU_Set, EXPR_Set, VA_Set to data/AffWild2/data/annotations folder
   2. Extract ccropped_aligned.zip to data/AffWild2/data/cropped_aligned folder
   3. Extract ccropped.zip and merge batch 1&2 folder to data/AffWild2/data/cropped folder
   4. Extract videos.zip and merge batch 1&2 folder to data/AffWild2/data/cropped_aligned folder
5. Download weight files and copy to folder submit1/weights from https://drive.google.com/drive/folders/1rJB2viPCxw93qFSaga3uqC6OfWMKRHn2?usp=sharing
6. Open JupyterLab and run *.ipynb in submit folder to output the results (
   • Run sel_t[xx].ipynb to output the prediction files(modify params parameter if neccessary)
   • Run sel_t[xx]_submit.ipynb to output the result folder (modify params parameter if neccessary)

Proposed model

Aff-Wild2 dataset

• Overview cropped_aligned image frames in different videos

• Overview cropped_aligned image frames in the same videos

• Data Distribution in Basic Emotion Recognition Track on Training and Validation

• Data Distribution in Valence-Arousal Regression Track on Training and Validation

Result

• List Models
  

• List Results
  

• Fusion Results on Validation: E xpr. Score = 0.533, Valence-Arousal Score = 0.5126
  

• Submission results: Track 1 Valence-Arousal Challenge on Validation: 0.484 (1), 0.534 (2), 0.514 (3), and 0.527 (4)
  Track 2 Basic Emotion Recognition Challenge on Validation: 0.501 (1), 0.492 (2), 0.478 (3), and 0.543 (4)
  

Baseline paper:

@misc{kollias2020analysing,
    title={Analysing Affective Behavior in the First ABAW 2020 Competition},
    author={Dimitrios Kollias and Attila Schulc and Elnar Hajiyev and Stefanos Zafeiriou},
    year={2020},
    eprint={2001.11409},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}