  Explainability is key to enhancing artificial intelligence's trustworthiness
in medicine. However, several issues remain concerning the actual benefit of
explainable models for clinical decision-making. Firstly, there is a lack of
consensus on an evaluation framework for quantitatively assessing the practical
benefits that effective explainability should provide to practitioners.
Secondly, physician-centered evaluations of explainability are limited.
Thirdly, the utility of built-in attention mechanisms in transformer-based
models as an explainability technique is unclear. We hypothesize that superior
attention maps should align with the information that physicians focus on,
potentially reducing prediction uncertainty and increasing model reliability.
We employed a multimodal transformer to predict lymph node metastasis in rectal
cancer using clinical data and magnetic resonance imaging, exploring how well
attention maps, visualized through a state-of-the-art technique, can achieve
agreement with physician understanding. We estimated the model's uncertainty
using meta-level information like prediction probability variance and
quantified agreement. Our assessment of whether this agreement reduces
uncertainty found no significant effect. In conclusion, this case study did not
confirm the anticipated benefit of attention maps in enhancing model
reliability. Superficial explanations could do more harm than good by
misleading physicians into relying on uncertain predictions, suggesting that
the current state of attention mechanisms in explainability should not be
overestimated. Identifying explainability mechanisms truly beneficial for
clinical decision-making remains essential.
