  Decision processes of computer vision models - especially deep neural
networks - are opaque in nature, meaning that these decisions cannot be
understood by humans. Thus, over the last years, many methods to provide
human-understandable explanations have been proposed. For image classification,
the most common group are saliency methods, which provide (super-)pixelwise
feature attribution scores for input images. But their evaluation still poses a
problem, as their results cannot be simply compared to the unknown ground
truth. To overcome this, a slew of different proxy metrics have been defined,
which are - as the explainability methods themselves - often built on intuition
and thus, are possibly unreliable. In this paper, new evaluation metrics for
saliency methods are developed and common saliency methods are benchmarked on
ImageNet. In addition, a scheme for reliability evaluation of such metrics is
proposed that is based on concepts from psychometric testing. The used code can
be found at
https://github.com/lelo204/ClassificationMetricsForImageExplanations .
