  Deep neural networks have been well-known for their superb handling of
various machine learning and artificial intelligence tasks. However, due to
their over-parameterized black-box nature, it is often difficult to understand
the prediction results of deep models. In recent years, many interpretation
tools have been proposed to explain or reveal how deep models make decisions.
In this paper, we review this line of research and try to make a comprehensive
survey. Specifically, we first introduce and clarify two basic concepts --
interpretations and interpretability -- that people usually get confused about.
To address the research efforts in interpretations, we elaborate the designs of
a number of interpretation algorithms, from different perspectives, by
proposing a new taxonomy. Then, to understand the interpretation results, we
also survey the performance metrics for evaluating interpretation algorithms.
Further, we summarize the current works in evaluating models' interpretability
using "trustworthy" interpretation algorithms. Finally, we review and discuss
the connections between deep models' interpretations and other factors, such as
adversarial robustness and learning from interpretations, and we introduce
several open-source libraries for interpretation algorithms and evaluation
approaches.
