  The prevalence of employing attention mechanisms has brought along concerns
on the interpretability of attention distributions. Although it provides
insights about how a model is operating, utilizing attention as the explanation
of model predictions is still highly dubious. The community is still seeking
more interpretable strategies for better identifying local active regions that
contribute the most to the final decision. To improve the interpretability of
existing attention models, we propose a novel Bilinear Representative
Non-Parametric Attention (BR-NPA) strategy that captures the task-relevant
human-interpretable information. The target model is first distilled to have
higher-resolution intermediate feature maps. From which, representative
features are then grouped based on local pairwise feature similarity, to
produce finer-grained, more precise attention maps highlighting task-relevant
parts of the input. The obtained attention maps are ranked according to the
activity level of the compound feature, which provides information regarding
the important level of the highlighted regions. The proposed model can be
easily adapted in a wide variety of modern deep models, where classification is
involved. Extensive quantitative and qualitative experiments showcase more
comprehensive and accurate visual explanations compared to state-of-the-art
attention models and visualizations methods across multiple tasks including
fine-grained image classification, few-shot classification, and person
re-identification, without compromising the classification accuracy. The
proposed visualization model sheds imperative light on how neural networks `pay
their attention' differently in different tasks.
