  Explainable AI has emerged to be a key component for black-box machine
learning approaches in domains with a high demand for reliability or
transparency. Examples are medical assistant systems, and applications
concerned with the General Data Protection Regulation of the European Union,
which features transparency as a cornerstone. Such demands require the ability
to audit the rationale behind a classifier's decision. While visualizations are
the de facto standard of explanations, they come short in terms of
expressiveness in many ways: They cannot distinguish between different
attribute manifestations of visual features (e.g. eye open vs. closed), and
they cannot accurately describe the influence of absence of, and relations
between features. An alternative would be more expressive symbolic surrogate
models. However, these require symbolic inputs, which are not readily available
in most computer vision tasks. In this paper we investigate how to overcome
this: We use inherent features learned by the network to build a global,
expressive, verbal explanation of the rationale of a feed-forward convolutional
deep neural network (DNN). The semantics of the features are mined by a concept
analysis approach trained on a set of human understandable visual concepts. The
explanation is found by an Inductive Logic Programming (ILP) method and
presented as first-order rules. We show that our explanation is faithful to the
original black-box model.
  The code for our experiments is available at
https://github.com/mc-lovin-mlem/concept-embeddings-and-ilp/tree/ki2020.
