  Recently, deep learning based facial expression recognition (FER) methods
have attracted considerable attention and they usually require large-scale
labelled training data. Nonetheless, the publicly available facial expression
databases typically contain a small amount of labelled data. In this paper, to
overcome the above issue, we propose a novel joint deep learning of facial
expression synthesis and recognition method for effective FER. More
specifically, the proposed method involves a two-stage learning procedure.
Firstly, a facial expression synthesis generative adversarial network (FESGAN)
is pre-trained to generate facial images with different facial expressions. To
increase the diversity of the training images, FESGAN is elaborately designed
to generate images with new identities from a prior distribution. Secondly, an
expression recognition network is jointly learned with the pre-trained FESGAN
in a unified framework. In particular, the classification loss computed from
the recognition network is used to simultaneously optimize the performance of
both the recognition network and the generator of FESGAN. Moreover, in order to
alleviate the problem of data bias between the real images and the synthetic
images, we propose an intra-class loss with a novel real data-guided
back-propagation (RDBP) algorithm to reduce the intra-class variations of
images from the same class, which can significantly improve the final
performance. Extensive experimental results on public facial expression
databases demonstrate the superiority of the proposed method compared with
several state-of-the-art FER methods.
