  We investigate how neural networks can learn and process languages with
hierarchical, compositional semantics. To this end, we define the artificial
task of processing nested arithmetic expressions, and study whether different
types of neural networks can learn to compute their meaning. We find that
recursive neural networks can find a generalising solution to this problem, and
we visualise this solution by breaking it up in three steps: project, sum and
squash. As a next step, we investigate recurrent neural networks, and show that
a gated recurrent unit, that processes its input incrementally, also performs
very well on this task. To develop an understanding of what the recurrent
network encodes, visualisation techniques alone do not suffice. Therefore, we
develop an approach where we formulate and test multiple hypotheses on the
information encoded and processed by the network. For each hypothesis, we
derive predictions about features of the hidden state representations at each
time step, and train 'diagnostic classifiers' to test those predictions. Our
results indicate that the networks follow a strategy similar to our
hypothesised 'cumulative strategy', which explains the high accuracy of the
network on novel expressions, the generalisation to longer expressions than
seen in training, and the mild deterioration with increasing length. This is
turn shows that diagnostic classifiers can be a useful technique for opening up
the black box of neural networks. We argue that diagnostic classification,
unlike most visualisation techniques, does scale up from small networks in a
toy domain, to larger and deeper recurrent networks dealing with real-life
data, and may therefore contribute to a better understanding of the internal
dynamics of current state-of-the-art models in natural language processing.
