  Despite excellent performance of deep neural networks (DNNs) in image
classification, detection, and prediction, characterizing how DNNs make a given
decision remains an open problem, resulting in a number of interpretability
methods. Post-hoc interpretability methods primarily aim to quantify the
importance of input features with respect to the class probabilities. However,
due to the lack of ground truth and the existence of interpretability methods
with diverse operating characteristics, evaluating these methods is a crucial
challenge. A popular approach to evaluate interpretability methods is to
perturb input features deemed important for a given prediction and observe the
decrease in accuracy. However, perturbation itself may introduce artifacts. We
propose a method for estimating the impact of such artifacts on the fidelity
estimation by utilizing model accuracy curves from perturbing input features
according to the Most Import First (MIF) and Least Import First (LIF) orders.
Using the ResNet-50 trained on the ImageNet, we demonstrate the proposed
fidelity estimation of four popular post-hoc interpretability methods.
