  Recent years have shown an increased development of methods for justifying
the predictions of neural networks through visual explanations. These
explanations usually take the form of heatmaps which assign a saliency (or
relevance) value to each pixel of the input image that expresses how relevant
the pixel is for the prediction of a label.
  Complementing this development, evaluation methods have been proposed to
assess the "goodness" of such explanations. On the one hand, some of these
methods rely on synthetic datasets. However, this introduces the weakness of
having limited guarantees regarding their applicability on more realistic
settings. On the other hand, some methods rely on metrics for objective
evaluation. However the level to which some of these evaluation methods perform
with respect to each other is uncertain.
  Taking this into account, we conduct a comprehensive study on a subset of the
ImageNet-1k validation set where we evaluate a number of different
commonly-used explanation methods following a set of evaluation methods. We
complement our study with sanity checks on the studied evaluation methods as a
means to investigate their reliability and the impact of characteristics of the
explanations on the evaluation methods.
  Results of our study suggest that there is a lack of coherency on the grading
provided by some of the considered evaluation methods. Moreover, we have
identified some characteristics of the explanations, e.g. sparsity, which can
have a significant effect on the performance.
