  The perturbation test remains the go-to evaluation approach for explanation
methods in computer vision. This evaluation method has a major drawback of
test-time distribution shift due to pixel-masking that is not present in the
training set. To overcome this drawback, we propose a novel evaluation
framework called \textbf{Inpainting the Gaps (InG)}. Specifically, we propose
inpainting parts that constitute partial or complete objects in an image. In
this way, one can perform meaningful image perturbations with lower test-time
distribution shifts, thereby improving the efficacy of the perturbation test.
InG is applied to the PartImageNet dataset to evaluate the performance of
popular explanation methods for three training strategies of the Vision
Transformer (ViT). Based on this evaluation, we found Beyond Intuition and
Generic Attribution to be the two most consistent explanation models. Further,
and interestingly, the proposed framework results in higher and more consistent
evaluation scores across all the ViT models considered in this work. To the
best of our knowledge, InG is the first semi-synthetic framework for the
evaluation of ViT explanation methods.
