  Deep neural networks have proved to be a very effective way to perform
classification tasks. They excel when the input data is high dimensional, the
relationship between the input and the output is complicated, and the number of
labeled training examples is large. But it is hard to explain why a learned
network makes a particular classification decision on a particular test case.
This is due to their reliance on distributed hierarchical representations. If
we could take the knowledge acquired by the neural net and express the same
knowledge in a model that relies on hierarchical decisions instead, explaining
a particular decision would be much easier. We describe a way of using a
trained neural net to create a type of soft decision tree that generalizes
better than one learned directly from the training data.
