  In computer vision, some attribution methods for explaining CNNs attempt to
study how the intermediate features affect the network prediction. However,
they usually ignore the feature hierarchies among the intermediate features.
This paper introduces a hierarchical decomposition framework to explain CNN's
decision-making process in a top-down manner. Specifically, we propose a
gradient-based activation propagation (gAP) module that can decompose any
intermediate CNN decision to its lower layers and find the supporting features.
Then we utilize the gAP module to iteratively decompose the network decision to
the supporting evidence from different CNN layers. The proposed framework can
generate a deep hierarchy of strongly associated supporting evidence for the
network decision, which provides insight into the decision-making process.
Moreover, gAP is effort-free for understanding CNN-based models without network
architecture modification and extra training process. Experiments show the
effectiveness of the proposed method. The code and interactive demo website
will be made publicly available.
