  Humans are able to explain their reasoning. On the contrary, deep neural
networks are not. This paper attempts to bridge this gap by introducing a new
way to design interpretable neural networks for classification, inspired by
physiological evidence of the human visual system's inner-workings. This paper
proposes a neural network design paradigm, termed InterpNET, which can be
combined with any existing classification architecture to generate natural
language explanations of the classifications. The success of the module relies
on the assumption that the network's computation and reasoning is represented
in its internal layer activations. While in principle InterpNET could be
applied to any existing classification architecture, it is evaluated via an
image classification and explanation task. Experiments on a CUB bird
classification and explanation dataset show qualitatively and quantitatively
that the model is able to generate high-quality explanations. While the current
state-of-the-art METEOR score on this dataset is 29.2, InterpNET achieves a
much higher METEOR score of 37.9.
