  Using attention weights to identify information that is important for models'
decision-making is a popular approach to interpret attention-based neural
networks. This is commonly realized in practice through the generation of a
heat-map for every single document based on attention weights. However, this
interpretation method is fragile, and easy to find contradictory examples. In
this paper, we propose a corpus-level explanation approach, which aims to
capture causal relationships between keywords and model predictions via
learning the importance of keywords for predicted labels across a training
corpus based on attention weights. Based on this idea, we further propose a
concept-based explanation method that can automatically learn higher-level
concepts and their importance to model prediction tasks. Our concept-based
explanation method is built upon a novel Abstraction-Aggregation Network, which
can automatically cluster important keywords during an end-to-end training
process. We apply these methods to the document classification task and show
that they are powerful in extracting semantically meaningful keywords and
concepts. Our consistency analysis results based on an attention-based Na\"ive
Bayes classifier also demonstrate these keywords and concepts are important for
model predictions.
