  A major challenge in Explainable AI is in correctly interpreting activations
of hidden neurons: accurate interpretations would provide insights into the
question of what a deep learning system has internally detected as relevant on
the input, demystifying the otherwise black-box character of deep learning
systems. The state of the art indicates that hidden node activations can, in
some cases, be interpretable in a way that makes sense to humans, but
systematic automated methods that would be able to hypothesize and verify
interpretations of hidden neuron activations are underexplored. In this paper,
we provide such a method and demonstrate that it provides meaningful
interpretations. Our approach is based on using large-scale background
knowledge approximately 2 million classes curated from the Wikipedia concept
hierarchy together with a symbolic reasoning approach called Concept Induction
based on description logics, originally developed for applications in the
Semantic Web field. Our results show that we can automatically attach
meaningful labels from the background knowledge to individual neurons in the
dense layer of a Convolutional Neural Network through a hypothesis and
verification process.
