  Deep learning has shown superb performance in detecting objects and
classifying images, ensuring a great promise for analyzing medical imaging.
Translating the success of deep learning to medical imaging, in which doctors
need to understand the underlying process, requires the capability to interpret
and explain the prediction of neural networks. Interpretability of deep neural
networks often relies on estimating the importance of input features (e.g.,
pixels) with respect to the outcome (e.g., class probability). However, a
number of importance estimators (also known as saliency maps) have been
developed and it is unclear which ones are more relevant for medical imaging
applications. In the present work, we investigated the performance of several
importance estimators in explaining the classification of computed tomography
(CT) images by a convolutional deep network, using three distinct evaluation
metrics. First, the model-centric fidelity measures a decrease in the model
accuracy when certain inputs are perturbed. Second, concordance between
importance scores and the expert-defined segmentation masks is measured on a
pixel level by a receiver operating characteristic (ROC) curves. Third, we
measure a region-wise overlap between a XRAI-based map and the segmentation
mask by Dice Similarity Coefficients (DSC). Overall, two versions of SmoothGrad
topped the fidelity and ROC rankings, whereas both Integrated Gradients and
SmoothGrad excelled in DSC evaluation. Interestingly, there was a critical
discrepancy between model-centric (fidelity) and human-centric (ROC and DSC)
evaluation. Expert expectation and intuition embedded in segmentation maps does
not necessarily align with how the model arrived at its prediction.
Understanding this difference in interpretability would help harnessing the
power of deep learning in medicine.
