  Explanations are crucial parts of deep neural network (DNN) classifiers. In
high stakes applications, faithful and robust explanations are important to
understand and gain trust in DNN classifiers. However, recent work has shown
that state-of-the-art attribution methods in text classifiers are susceptible
to imperceptible adversarial perturbations that alter explanations
significantly while maintaining the correct prediction outcome. If undetected,
this can critically mislead the users of DNNs. Thus, it is crucial to
understand the influence of such adversarial perturbations on the networks'
explanations and their perceptibility. In this work, we establish a novel
definition of attribution robustness (AR) in text classification, based on
Lipschitz continuity. Crucially, it reflects both attribution change induced by
adversarial input alterations and perceptibility of such alterations. Moreover,
we introduce a wide set of text similarity measures to effectively capture
locality between two text samples and imperceptibility of adversarial
perturbations in text. We then propose our novel TransformerExplanationAttack
(TEA), a strong adversary that provides a tight estimation for attribution
robustness in text classification. TEA uses state-of-the-art language models to
extract word substitutions that result in fluent, contextual adversarial
samples. Finally, with experiments on several text classification
architectures, we show that TEA consistently outperforms current
state-of-the-art AR estimators, yielding perturbations that alter explanations
to a greater extent while being more fluent and less perceptible.
