  We present a method for neural network interpretability by combining feature
attribution with counterfactual explanations to generate attribution maps that
highlight the most discriminative features between pairs of classes. We show
that this method can be used to quantitatively evaluate the performance of
feature attribution methods in an objective manner, thus preventing potential
observer bias. We evaluate the proposed method on three diverse datasets,
including a challenging artificial dataset and real-world biological data. We
show quantitatively and qualitatively that the highlighted features are
substantially more discriminative than those extracted using conventional
attribution methods and argue that this type of explanation is better suited
for understanding fine grained class differences as learned by a deep neural
network.
