  Post-hoc gradient-based interpretability methods [Simonyan et al., 2013,
Smilkov et al., 2017] that provide instance-specific explanations of model
predictions are often based on assumption (A): magnitude of input gradients --
gradients of logits with respect to input -- noisily highlight discriminative
task-relevant features. In this work, we test the validity of assumption (A)
using a three-pronged approach. First, we develop an evaluation framework,
DiffROAR, to test assumption (A) on four image classification benchmarks. Our
results suggest that (i) input gradients of standard models (i.e., trained on
original data) may grossly violate (A), whereas (ii) input gradients of
adversarially robust models satisfy (A). Second, we introduce BlockMNIST, an
MNIST-based semi-real dataset, that by design encodes a priori knowledge of
discriminative features. Our analysis on BlockMNIST leverages this information
to validate as well as characterize differences between input gradient
attributions of standard and robust models. Finally, we theoretically prove
that our empirical findings hold on a simplified version of the BlockMNIST
dataset. Specifically, we prove that input gradients of standard
one-hidden-layer MLPs trained on this dataset do not highlight
instance-specific signal coordinates, thus grossly violating assumption (A).
Our findings motivate the need to formalize and test common assumptions in
interpretability in a falsifiable manner [Leavitt and Morcos, 2020]. We believe
that the DiffROAR evaluation framework and BlockMNIST-based datasets can serve
as sanity checks to audit instance-specific interpretability methods; code and
data available at https://github.com/harshays/inputgradients.
