  While the evaluation of explanations is an important step towards trustworthy
models, it needs to be done carefully, and the employed metrics need to be
well-understood. Specifically model randomization testing is often
overestimated and regarded as a sole criterion for selecting or discarding
certain explanation methods. To address shortcomings of this test, we start by
observing an experimental gap in the ranking of explanation methods between
randomization-based sanity checks [1] and model output faithfulness measures
(e.g. [25]). We identify limitations of model-randomization-based sanity checks
for the purpose of evaluating explanations. Firstly, we show that uninformative
attribution maps created with zero pixel-wise covariance easily achieve high
scores in this type of checks. Secondly, we show that top-down model
randomization preserves scales of forward pass activations with high
probability. That is, channels with large activations have a high probility to
contribute strongly to the output, even after randomization of the network on
top of them. Hence, explanations after randomization can only be expected to
differ to a certain extent. This explains the observed experimental gap. In
summary, these results demonstrate the inadequacy of model-randomization-based
sanity checks as a criterion to rank attribution methods.
