  In recent decades, 3D morphable model (3DMM) has been commonly used in
image-based photorealistic 3D face reconstruction. However, face images are
often corrupted by serious occlusion by non-face objects including eyeglasses,
masks, and hands. Such objects block the correct capture of landmarks and
shading information. Therefore, the reconstructed 3D face model is hardly
reusable. In this paper, a novel method is proposed to restore de-occluded face
images based on inverse use of 3DMM and generative adversarial network. We
utilize the 3DMM prior to the proposed adversarial network and combine a global
and local adversarial convolutional neural network to learn face de-occlusion
model. The 3DMM serves not only as geometric prior but also proposes the face
region for the local discriminator. Experiment results confirm the
effectiveness and robustness of the proposed algorithm in removing challenging
types of occlusions with various head poses and illumination. Furthermore, the
proposed method reconstructs the correct 3D face model with de-occluded
textures.
