  Interpretation of deep learning remains a very challenging problem. Although
the Class Activation Map (CAM) is widely used to interpret deep model
predictions by highlighting object location, it fails to provide insight into
the salient features used by the model to make decisions. Furthermore, existing
evaluation protocols often overlook the correlation between interpretability
performance and the model's decision quality, which presents a more fundamental
issue. This paper proposes a new two-stage interpretability method called the
Decomposition Class Activation Map (Decom-CAM), which offers a feature-level
interpretation of the model's prediction. Decom-CAM decomposes intermediate
activation maps into orthogonal features using singular value decomposition and
generates saliency maps by integrating them. The orthogonality of features
enables CAM to capture local features and can be used to pinpoint semantic
components such as eyes, noses, and faces in the input image, making it more
beneficial for deep model interpretation. To ensure a comprehensive comparison,
we introduce a new evaluation protocol by dividing the dataset into subsets
based on classification accuracy results and evaluating the interpretability
performance on each subset separately. Our experiments demonstrate that the
proposed Decom-CAM outperforms current state-of-the-art methods significantly
by generating more precise saliency maps across all levels of classification
accuracy. Combined with our feature-level interpretability approach, this paper
could pave the way for a new direction for understanding the decision-making
process of deep neural networks.
