  Providing explanations for deep neural network (DNN) models is crucial for
their use in security-sensitive domains. A plethora of interpretation models
have been proposed to help users understand the inner workings of DNNs: how
does a DNN arrive at a specific decision for a given input? The improved
interpretability is believed to offer a sense of security by involving human in
the decision-making process. Yet, due to its data-driven nature, the
interpretability itself is potentially susceptible to malicious manipulations,
about which little is known thus far.
  Here we bridge this gap by conducting the first systematic study on the
security of interpretable deep learning systems (IDLSes). We show that existing
\imlses are highly vulnerable to adversarial manipulations. Specifically, we
present ADV^2, a new class of attacks that generate adversarial inputs not only
misleading target DNNs but also deceiving their coupled interpretation models.
Through empirical evaluation against four major types of IDLSes on benchmark
datasets and in security-critical applications (e.g., skin cancer diagnosis),
we demonstrate that with ADV^2 the adversary is able to arbitrarily designate
an input's prediction and interpretation. Further, with both analytical and
empirical evidence, we identify the prediction-interpretation gap as one root
cause of this vulnerability -- a DNN and its interpretation model are often
misaligned, resulting in the possibility of exploiting both models
simultaneously. Finally, we explore potential countermeasures against ADV^2,
including leveraging its low transferability and incorporating it in an
adversarial training framework. Our findings shed light on designing and
operating IDLSes in a more secure and informative fashion, leading to several
promising research directions.
