  Manipulating facial expressions is a challenging task due to fine-grained
shape changes produced by facial muscles and the lack of input-output pairs for
supervised learning. Unlike previous methods using Generative Adversarial
Networks (GAN), which rely on cycle-consistency loss or sparse geometry
(landmarks) loss for expression synthesis, we propose a novel GAN framework to
exploit 3D dense (depth and surface normals) information for expression
manipulation. However, a large-scale dataset containing RGB images with
expression annotations and their corresponding depth maps is not available. To
this end, we propose to use an off-the-shelf state-of-the-art 3D reconstruction
model to estimate the depth and create a large-scale RGB-Depth dataset after a
manual data clean-up process. We utilise this dataset to minimise the novel
depth consistency loss via adversarial learning (note we do not have ground
truth depth maps for generated face images) and the depth categorical loss of
synthetic data on the discriminator. In addition, to improve the generalisation
and lower the bias of the depth parameters, we propose to use a novel
confidence regulariser on the discriminator side of the framework. We
extensively performed both quantitative and qualitative evaluations on two
publicly available challenging facial expression benchmarks: AffectNet and
RaFD. Our experiments demonstrate that the proposed method outperforms the
competitive baseline and existing arts by a large margin.
