  The evolution of deep learning and artificial intelligence has significantly
reshaped technological landscapes. However, their effective application in
crucial sectors such as medicine demands more than just superior performance,
but trustworthiness as well. While interpretability plays a pivotal role,
existing explainable AI (XAI) approaches often do not reveal {\em Clever Hans}
behavior where a model makes (ungeneralizable) correct predictions using
spurious correlations or biases in data. Likewise, current post-hoc XAI methods
are susceptible to generating unjustified counterfactual examples. In this
paper, we approach XAI with an innovative {\em model debugging} methodology
realized through Jacobian Saliency Map (JSM). To cast the problem into a
concrete context, we employ Alzheimer's disease (AD) diagnosis as the use case,
motivated by its significant impact on human lives and the formidable challenge
in its early detection, stemming from the intricate nature of its progression.
We introduce an interpretable, multimodal model for AD classification over its
multi-stage progression, incorporating JSM as a modality-agnostic tool that
provides insights into volumetric changes indicative of brain abnormalities.
Our extensive evaluation including ablation study manifests the efficacy of
using JSM for model debugging and interpretation, while significantly enhancing
model accuracy as well.
