  Artificial intelligence (AI) has huge potential to improve the health and
well-being of people, but adoption in clinical practice is still limited. Lack
of transparency is identified as one of the main barriers to implementation, as
clinicians should be confident the AI system can be trusted. Explainable AI has
the potential to overcome this issue and can be a step towards trustworthy AI.
In this paper we review the recent literature to provide guidance to
researchers and practitioners on the design of explainable AI systems for the
health-care domain and contribute to formalization of the field of explainable
AI. We argue the reason to demand explainability determines what should be
explained as this determines the relative importance of the properties of
explainability (i.e. interpretability and fidelity). Based on this, we propose
a framework to guide the choice between classes of explainable AI methods
(explainable modelling versus post-hoc explanation; model-based,
attribution-based, or example-based explanations; global and local
explanations). Furthermore, we find that quantitative evaluation metrics, which
are important for objective standardized evaluation, are still lacking for some
properties (e.g. clarity) and types of explanations (e.g. example-based
methods). We conclude that explainable modelling can contribute to trustworthy
AI, but the benefits of explainability still need to be proven in practice and
complementary measures might be needed to create trustworthy AI in health care
(e.g. reporting data quality, performing extensive (external) validation, and
regulation).
