  Integrated interpretability without sacrificing the prediction accuracy of
decision making algorithms has the potential of greatly improving their value
to the user. Instead of assigning a label to an image directly, we propose to
learn iterative binary sub-decisions, inducing sparsity and transparency in the
decision making process. The key aspect of our model is its ability to build a
decision tree whose structure is encoded into the memory representation of a
Recurrent Neural Network jointly learned by two models communicating through
message passing. In addition, our model assigns a semantic meaning to each
decision in the form of binary attributes, providing concise, semantic and
relevant rationalizations to the user. On three benchmark image classification
datasets, including the large-scale ImageNet, our model generates human
interpretable binary decision sequences explaining the predictions of the
network while maintaining state-of-the-art accuracy.
