  Current benchmarks for facial expression recognition (FER) mainly focus on
static images, while there are limited datasets for FER in videos. It is still
ambiguous to evaluate whether performances of existing methods remain
satisfactory in real-world application-oriented scenes. For example, the
"Happy" expression with high intensity in Talk-Show is more discriminating than
the same expression with low intensity in Official-Event. To fill this gap, we
build a large-scale multi-scene dataset, coined as FERV39k. We analyze the
important ingredients of constructing such a novel dataset in three aspects:
(1) multi-scene hierarchy and expression class, (2) generation of candidate
video clips, (3) trusted manual labelling process. Based on these guidelines,
we select 4 scenarios subdivided into 22 scenes, annotate 86k samples
automatically obtained from 4k videos based on the well-designed workflow, and
finally build 38,935 video clips labeled with 7 classic expressions. Experiment
benchmarks on four kinds of baseline frameworks were also provided and further
analysis on their performance across different scenes and some challenges for
future research were given. Besides, we systematically investigate key
components of DFER by ablation studies. The baseline framework and our project
will be available.
