  Image inpainting is an essential task for multiple practical applications
like object removal and image editing. Deep GAN-based models greatly improve
the inpainting performance in structures and textures within the hole, but
might also generate unexpected artifacts like broken structures or color blobs.
Users perceive these artifacts to judge the effectiveness of inpainting models,
and retouch these imperfect areas to inpaint again in a typical retouching
workflow. Inspired by this workflow, we propose a new learning task of
automatic segmentation of inpainting perceptual artifacts, and apply the model
for inpainting model evaluation and iterative refinement. Specifically, we
first construct a new inpainting artifacts dataset by manually annotating
perceptual artifacts in the results of state-of-the-art inpainting models. Then
we train advanced segmentation networks on this dataset to reliably localize
inpainting artifacts within inpainted images. Second, we propose a new
interpretable evaluation metric called Perceptual Artifact Ratio (PAR), which
is the ratio of objectionable inpainted regions to the entire inpainted area.
PAR demonstrates a strong correlation with real user preference. Finally, we
further apply the generated masks for iterative image inpainting by combining
our approach with multiple recent inpainting methods. Extensive experiments
demonstrate the consistent decrease of artifact regions and inpainting quality
improvement across the different methods.
