  Medical experts may use Artificial Intelligence (AI) systems with greater
trust if these are supported by contextual explanations that let the
practitioner connect system inferences to their context of use. However, their
importance in improving model usage and understanding has not been extensively
studied. Hence, we consider a comorbidity risk prediction scenario and focus on
contexts regarding the patients clinical state, AI predictions about their risk
of complications, and algorithmic explanations supporting the predictions. We
explore how relevant information for such dimensions can be extracted from
Medical guidelines to answer typical questions from clinical practitioners. We
identify this as a question answering (QA) task and employ several
state-of-the-art LLMs to present contexts around risk prediction model
inferences and evaluate their acceptability. Finally, we study the benefits of
contextual explanations by building an end-to-end AI pipeline including data
cohorting, AI risk modeling, post-hoc model explanations, and prototyped a
visual dashboard to present the combined insights from different context
dimensions and data sources, while predicting and identifying the drivers of
risk of Chronic Kidney Disease - a common type-2 diabetes comorbidity. All of
these steps were performed in engagement with medical experts, including a
final evaluation of the dashboard results by an expert medical panel. We show
that LLMs, in particular BERT and SciBERT, can be readily deployed to extract
some relevant explanations to support clinical usage. To understand the
value-add of the contextual explanations, the expert panel evaluated these
regarding actionable insights in the relevant clinical setting. Overall, our
paper is one of the first end-to-end analyses identifying the feasibility and
benefits of contextual explanations in a real-world clinical use case.
