  The lack of transparency and explainability hinders the clinical adoption of
Machine learning (ML) algorithms. While explainable artificial intelligence
(XAI) methods have been proposed, little research has focused on the agreement
between these methods and expert clinical knowledge. This study applies current
state-of-the-art explainability methods to clinical decision support algorithms
developed for Electronic Medical Records (EMR) data to analyse the concordance
between these factors and discusses causes for identified discrepancies from a
clinical and technical perspective. Important factors for achieving trustworthy
XAI solutions for clinical decision support are also discussed.
