  Facial Expression Recognition (FER) in the wild is extremely challenging due
to occlusions, variant head poses, face deformation and motion blur under
unconstrained conditions. Although substantial progresses have been made in
automatic FER in the past few decades, previous studies were mainly designed
for lab-controlled FER. Real-world occlusions, variant head poses and other
issues definitely increase the difficulty of FER on account of these
information-deficient regions and complex backgrounds. Different from previous
pure CNNs based methods, we argue that it is feasible and practical to
translate facial images into sequences of visual words and perform expression
recognition from a global perspective. Therefore, we propose the Visual
Transformers with Feature Fusion (VTFF) to tackle FER in the wild by two main
steps. First, we propose the attentional selective fusion (ASF) for leveraging
two kinds of feature maps generated by two-branch CNNs. The ASF captures
discriminative information by fusing multiple features with the global-local
attention. The fused feature maps are then flattened and projected into
sequences of visual words. Second, inspired by the success of Transformers in
natural language processing, we propose to model relationships between these
visual words with the global self-attention. The proposed method is evaluated
on three public in-the-wild facial expression datasets (RAF-DB, FERPlus and
AffectNet). Under the same settings, extensive experiments demonstrate that our
method shows superior performance over other methods, setting new state of the
art on RAF-DB with 88.14%, FERPlus with 88.81% and AffectNet with 61.85%. The
cross-dataset evaluation on CK+ shows the promising generalization capability
of the proposed method.
