  Although machine learning (ML) models of AI achieve high performances in
medicine, they are not free of errors. Empowering clinicians to identify
incorrect model recommendations is crucial for engendering trust in medical AI.
Explainable AI (XAI) aims to address this requirement by clarifying AI
reasoning to support the end users. Several studies on biomedical imaging
achieved promising results recently. Nevertheless, solutions for models using
tabular data are not sufficient to meet the requirements of clinicians yet.
This paper proposes a methodology to support clinicians in identifying failures
of ML models trained with tabular data. We built our methodology on three main
pillars: decomposing the feature set by leveraging clinical context latent
space, assessing the clinical association of global explanations, and Latent
Space Similarity (LSS) based local explanations. We demonstrated our
methodology on ML-based recognition of preterm infant morbidities caused by
infection. The risk of mortality, lifelong disability, and antibiotic
resistance due to model failures was an open research question in this domain.
We achieved to identify misclassification cases of two models with our
approach. By contextualizing local explanations, our solution provides
clinicians with actionable insights to support their autonomy for informed
final decisions.
