  Building facial analysis systems that generalize to extreme variations in
lighting and facial expressions is a challenging problem that can potentially
be alleviated using natural-looking synthetic data. Towards that, we propose
LEGAN, a novel synthesis framework that leverages perceptual quality judgments
for jointly manipulating lighting and expressions in face images, without
requiring paired training data. LEGAN disentangles the lighting and expression
subspaces and performs transformations in the feature space before upscaling to
the desired output image. The fidelity of the synthetic image is further
refined by integrating a perceptual quality estimation model, trained with face
images rendered using multiple synthesis methods and their crowd-sourced
naturalness ratings, into the LEGAN framework as an auxiliary discriminator.
Using objective metrics like FID and LPIPS, LEGAN is shown to generate higher
quality face images when compared with popular GAN models like StarGAN and
StarGAN-v2 for lighting and expression synthesis. We also conduct a perceptual
study using images synthesized by LEGAN and other GAN models and show the
correlation between our quality estimation and visual fidelity. Finally, we
demonstrate the effectiveness of LEGAN as training data augmenter for
expression recognition and face verification tasks.
