  Model explainability is essential for the creation of trustworthy Machine
Learning models in healthcare. An ideal explanation resembles the
decision-making process of a domain expert and is expressed using concepts or
terminology that is meaningful to the clinicians. To provide such an
explanation, we first associate the hidden units of the classifier to
clinically relevant concepts. We take advantage of radiology reports
accompanying the chest X-ray images to define concepts. We discover sparse
associations between concepts and hidden units using a linear sparse logistic
regression. To ensure that the identified units truly influence the
classifier's outcome, we adopt tools from Causal Inference literature and, more
specifically, mediation analysis through counterfactual interventions. Finally,
we construct a low-depth decision tree to translate all the discovered concepts
into a straightforward decision rule, expressed to the radiologist. We
evaluated our approach on a large chest x-ray dataset, where our model produces
a global explanation consistent with clinical knowledge.
