  Being able to explain the prediction to clinical end-users is a necessity to
leverage the power of AI models for clinical decision support. For medical
images, saliency maps are the most common form of explanation. The maps
highlight important features for AI model's prediction. Although many saliency
map methods have been proposed, it is unknown how well they perform on
explaining decisions on multi-modal medical images, where each modality/channel
carries distinct clinical meanings of the same underlying biomedical
phenomenon. Understanding such modality-dependent features is essential for
clinical users' interpretation of AI decisions. To tackle this clinically
important but technically ignored problem, we propose the MSFI
(Modality-Specific Feature Importance) metric to examine whether saliency maps
can highlight modality-specific important features. MSFI encodes the clinical
requirements on modality prioritization and modality-specific feature
localization. Our evaluations on 16 commonly used saliency map methods,
including a clinician user study, show that although most saliency map methods
captured modality importance information in general, most of them failed to
highlight modality-specific important features consistently and precisely. The
evaluation results guide the choices of saliency map methods and provide
insights to propose new ones targeting clinical applications.
