  This survey presents an overview of integrating prior knowledge into machine
learning systems in order to improve explainability. The complexity of machine
learning models has elicited research to make them more explainable. However,
most explainability methods cannot provide insight beyond the given data,
requiring additional information about the context. We propose to harness prior
knowledge to improve upon the explanation capabilities of machine learning
models. In this paper, we present a categorization of current research into
three main categories which either integrate knowledge into the machine
learning pipeline, into the explainability method or derive knowledge from
explanations. To classify the papers, we build upon the existing taxonomy of
informed machine learning and extend it from the perspective of explainability.
We conclude with open challenges and research directions.
