  Understanding the decisions of deep learning (DL) models is essential for the
acceptance of DL to risk-sensitive applications. Although methods, like class
activation maps (CAMs), give a glimpse into the black box, they do miss some
crucial information, thereby limiting its interpretability and merely providing
the considered locations of objects. To provide more insight into the models
and the influence of datasets, we propose an integrated feature analysis
method, which consists of feature distribution analysis and feature
decomposition, to look closer into the intermediate features extracted by DL
models. This integrated feature analysis could provide information on
overfitting, confounders, outliers in datasets, model redundancies and
principal features extracted by the models, and provide distribution
information to form a common intensity scale, which are missing in current CAM
algorithms. The integrated feature analysis was applied to eight different
datasets for general validation: photographs of handwritten digits, two
datasets of natural images and five medical datasets, including skin
photography, ultrasound, CT, X-rays and MRIs. The method was evaluated by
calculating the consistency between the CAMs average class activation levels
and the logits of the model. Based on the eight datasets, the correlation
coefficients through our method were all very close to 100%, and based on the
feature decomposition, 5%-25% of features could generate equally informative
saliency maps and obtain the same model performances as using all features.
This proves the reliability of the integrated feature analysis. As the proposed
methods rely on very few assumptions, this is a step towards better model
interpretation and a useful extension to existing CAM algorithms. Codes:
https://github.com/YanliLi27/IFA
