  Existing face completion solutions are primarily driven by end-to-end models
that directly generate 2D completions of 2D masked faces. By having to
implicitly account for geometric and photometric variations in facial shape and
appearance, such approaches result in unrealistic completions, especially under
large variations in pose, shape, illumination and mask sizes. To alleviate
these limitations, we introduce 3DFaceFill, an analysis-by-synthesis approach
for face completion that explicitly considers the image formation process. It
comprises three components, (1) an encoder that disentangles the face into its
constituent 3D mesh, 3D pose, illumination and albedo factors, (2) an
autoencoder that inpaints the UV representation of facial albedo, and (3) a
renderer that resynthesizes the completed face. By operating on the UV
representation, 3DFaceFill affords the power of correspondence and allows us to
naturally enforce geometrical priors (e.g. facial symmetry) more effectively.
Quantitatively, 3DFaceFill improves the state-of-the-art by up to 4dB higher
PSNR and 25% better LPIPS for large masks. And, qualitatively, it leads to
demonstrably more photorealistic face completions over a range of masks and
occlusions while preserving consistency in global and component-wise shape,
pose, illumination and eye-gaze.
