  Why do explainable AI (XAI) explanations in radiology, despite their promise
of transparency, still fail to gain human trust? Current XAI approaches provide
justification for predictions, however, these do not meet practitioners' needs.
These XAI explanations lack intuitive coverage of the evidentiary basis for a
given classification, posing a significant barrier to adoption. We posit that
XAI explanations that mirror human processes of reasoning and justification
with evidence may be more useful and trustworthy than traditional visual
explanations like heat maps. Using a radiology case study, we demonstrate how
radiology practitioners get other practitioners to see a diagnostic
conclusion's validity. Machine-learned classifications lack this evidentiary
grounding and consequently fail to elicit trust and adoption by potential
users. Insights from this study may generalize to guiding principles for
human-centered explanation design based on human reasoning and justification of
evidence.
