  The success of neural networks builds to a large extent on their ability to
create internal knowledge representations from real-world high-dimensional
data, such as images, sound, or text. Approaches to extract and present these
representations, in order to explain the neural network's decisions, is an
active and multifaceted research field. To gain a deeper understanding of a
central aspect of this field, we have performed a targeted review focusing on
research that aims to associate internal representations with human
understandable concepts. In doing this, we added a perspective on the existing
research by using primarily deductive nomological explanations as a proposed
taxonomy. We find this taxonomy and theories of causality, useful for
understanding what can be expected, and not expected, from neural network
explanations. The analysis additionally uncovers an ambiguity in the reviewed
literature related to the goal of model explainability; is it understanding the
ML model or, is it actionable explanations useful in the deployment domain?
