  Neural network models are widely used in a variety of domains, often as
black-box solutions, since they are not directly interpretable for humans. The
field of explainable artificial intelligence aims at developing explanation
methods to address this challenge, and several approaches have been developed
over the recent years, including methods for investigating what type of
knowledge these models internalise during the training process. Among these,
the method of concept detection, investigates which \emph{concepts} neural
network models learn to represent in order to complete their tasks. In this
work, we present an extension to the method of concept detection, named
\emph{concept backpropagation}, which provides a way of analysing how the
information representing a given concept is internalised in a given neural
network model. In this approach, the model input is perturbed in a manner
guided by a trained concept probe for the described model, such that the
concept of interest is maximised. This allows for the visualisation of the
detected concept directly in the input space of the model, which in turn makes
it possible to see what information the model depends on for representing the
described concept. We present results for this method applied to a various set
of input modalities, and discuss how our proposed method can be used to
visualise what information trained concept probes use, and the degree as to
which the representation of the probed concept is entangled within the neural
network model itself.
