  Clinical Decision Support Systems (CDSS) utilize evidence-based knowledge and
patient data to offer real-time recommendations, with Large Language Models
(LLMs) emerging as a promising tool to generate plain-text explanations for
medical decisions. This study explores the effectiveness and reliability of
LLMs in generating explanations for diagnoses based on patient complaints.
Three experienced doctors evaluated LLM-generated explanations of the
connection between patient complaints and doctor and model-assigned diagnoses
across several stages. Experimental results demonstrated that LLM explanations
significantly increased doctors' agreement rates with given diagnoses and
highlighted potential errors in LLM outputs, ranging from 5% to 30%. The study
underscores the potential and challenges of LLMs in healthcare and emphasizes
the need for careful integration and evaluation to ensure patient safety and
optimal clinical utility.
