  Deep neural networks are very successful on many vision tasks, but hard to
interpret due to their black box nature. To overcome this, various post-hoc
attribution methods have been proposed to identify image regions most
influential to the models' decisions. Evaluating such methods is challenging
since no ground truth attributions exist. We thus propose three novel
evaluation schemes to more reliably measure the faithfulness of those methods,
to make comparisons between them more fair, and to make visual inspection more
systematic. To address faithfulness, we propose a novel evaluation setting
(DiFull) in which we carefully control which parts of the input can influence
the output in order to distinguish possible from impossible attributions. To
address fairness, we note that different methods are applied at different
layers, which skews any comparison, and so evaluate all methods on the same
layers (ML-Att) and discuss how this impacts their performance on quantitative
metrics. For more systematic visualizations, we propose a scheme (AggAtt) to
qualitatively evaluate the methods on complete datasets. We use these
evaluation schemes to study strengths and shortcomings of some widely used
attribution methods. Finally, we propose a post-processing smoothing step that
significantly improves the performance of some attribution methods, and discuss
its applicability.
