  Studies have proven that domain bias and label bias exist in different Facial
Expression Recognition (FER) datasets, making it hard to improve the
performance of a specific dataset by adding other datasets. For the FER bias
issue, recent researches mainly focus on the cross-domain issue with advanced
domain adaption algorithms. This paper addresses another problem: how to boost
FER performance by leveraging cross-domain datasets. Unlike the coarse and
biased expression label, the facial Action Unit (AU) is fine-grained and
objective suggested by psychological studies. Motivated by this, we resort to
the AU information of different FER datasets for performance boosting and make
contributions as follows. First, we experimentally show that the naive joint
training of multiple FER datasets is harmful to the FER performance of
individual datasets. We further introduce expression-specific mean images and
AU cosine distances to measure FER dataset bias. This novel measurement shows
consistent conclusions with experimental degradation of joint training. Second,
we propose a simple yet conceptually-new framework, AU-aware Vision Transformer
(AU-ViT). It improves the performance of individual datasets by jointly
training auxiliary datasets with AU or pseudo-AU labels. We also find that the
AU-ViT is robust to real-world occlusions. Moreover, for the first time, we
prove that a carefully-initialized ViT achieves comparable performance to
advanced deep convolutional networks. Our AU-ViT achieves state-of-the-art
performance on three popular datasets, namely 91.10% on RAF-DB, 65.59% on
AffectNet, and 90.15% on FERPlus. The code and models will be released soon.
