  This paper aims to quantitatively explain rationales of each prediction that
is made by a pre-trained convolutional neural network (CNN). We propose to
learn a decision tree, which clarifies the specific reason for each prediction
made by the CNN at the semantic level. I.e., the decision tree decomposes
feature representations in high conv-layers of the CNN into elementary concepts
of object parts. In this way, the decision tree tells people which object parts
activate which filters for the prediction and how much they contribute to the
prediction score. Such semantic and quantitative explanations for CNN
predictions have specific values beyond the traditional pixel-level analysis of
CNNs. More specifically, our method mines all potential decision modes of the
CNN, where each mode represents a common case of how the CNN uses object parts
for prediction. The decision tree organizes all potential decision modes in a
coarse-to-fine manner to explain CNN predictions at different fine-grained
levels. Experiments have demonstrated the effectiveness of the proposed method.
