  3DMM conditioned face generation has gained traction due to its well-defined
controllability; however, the trade-off is lower sample quality: Previous works
such as DiscoFaceGAN and 3D-FM GAN show a significant FID gap compared to the
unconditional StyleGAN, suggesting that there is a quality tax to pay for
controllability. In this paper, we challenge the assumption that quality and
controllability cannot coexist. To pinpoint the previous issues, we
mathematically formalize the problem of 3DMM conditioned face generation. Then,
we devise simple solutions to the problem under our proposed framework. This
results in a new model that effectively removes the quality tax between 3DMM
conditioned face GANs and the unconditional StyleGAN.
