  Automatic facial action unit (AU) recognition is a challenging task due to
the scarcity of manual annotations. To alleviate this problem, a large amount
of efforts has been dedicated to exploiting various methods which leverage
numerous unlabeled data. However, many aspects with regard to some unique
properties of AUs, such as the regional and relational characteristics, are not
sufficiently explored in previous works. Motivated by this, we take the AU
properties into consideration and propose two auxiliary AU related tasks to
bridge the gap between limited annotations and the model performance in a
self-supervised manner via the unlabeled data. Specifically, to enhance the
discrimination of regional features with AU relation embedding, we design a
task of RoI inpainting to recover the randomly cropped AU patches. Meanwhile, a
single image based optical flow estimation task is proposed to leverage the
dynamic change of facial muscles and encode the motion information into the
global feature representation. Based on these two self-supervised auxiliary
tasks, local features, mutual relation and motion cues of AUs are better
captured in the backbone network with the proposed regional and temporal based
auxiliary task learning (RTATL) framework. Extensive experiments on BP4D and
DISFA demonstrate the superiority of our method and new state-of-the-art
performances are achieved.
