  Our interest in this paper is in the construction of symbolic explanations
for predictions made by a deep neural network. We will focus attention on deep
relational machines (DRMs, first proposed by H. Lodhi). A DRM is a deep network
in which the input layer consists of Boolean-valued functions (features) that
are defined in terms of relations provided as domain, or background, knowledge.
Our DRMs differ from those proposed by Lodhi, which use an Inductive Logic
Programming (ILP) engine to first select features (we use random selections
from a space of features that satisfies some approximate constraints on logical
relevance and non-redundancy). But why do the DRMs predict what they do? One
way of answering this is the LIME setting, in which readable proxies for a
black-box predictor. The proxies are intended only to model the predictions of
the black-box in local regions of the instance-space. But readability alone may
not enough: to be understandable, the local models must use relevant concepts
in an meaningful manner. We investigate the use of a Bayes-like approach to
identify logical proxies for local predictions of a DRM. We show: (a) DRM's
with our randomised propositionalization method achieve state-of-the-art
predictive performance; (b) Models in first-order logic can approximate the
DRM's prediction closely in a small local region; and (c) Expert-provided
relevance information can play the role of a prior to distinguish between
logical explanations that perform equivalently on prediction alone.
