  Quantitative evaluation has increased dramatically among recent video
inpainting work, but the video and mask content used to gauge performance has
received relatively little attention. Although attributes such as camera and
background scene motion inherently change the difficulty of the task and affect
methods differently, existing evaluation schemes fail to control for them,
thereby providing minimal insight into inpainting failure modes. To address
this gap, we propose the Diagnostic Evaluation of Video Inpainting on
Landscapes (DEVIL) benchmark, which consists of two contributions: (i) a novel
dataset of videos and masks labeled according to several key inpainting failure
modes, and (ii) an evaluation scheme that samples slices of the dataset
characterized by a fixed content attribute, and scores performance on each
slice according to reconstruction, realism, and temporal consistency quality.
By revealing systematic changes in performance induced by particular
characteristics of the input content, our challenging benchmark enables more
insightful analysis into video inpainting methods and serves as an invaluable
diagnostic tool for the field. Our code and data are available at
https://github.com/MichiganCOG/devil .
