  Not only are Deep Neural Networks (DNNs) black box models, but also we
frequently conceptualize them as such. We lack good interpretations of the
mechanisms linking inputs to outputs. Therefore, we find it difficult to
analyze in human-meaningful terms (1) what the network learned and (2) whether
the network learned. We present a hierarchical decomposition of the DNN
discrete classification map into logical (AND/OR) combinations of intermediate
(True/False) classifiers of the input. Those classifiers that can not be
further decomposed, called atoms, are (interpretable) linear classifiers. Taken
together, we obtain a logical circuit with linear classifier inputs that
computes the same label as the DNN. This circuit does not structurally resemble
the network architecture, and it may require many fewer parameters, depending
on the configuration of weights. In these cases, we obtain simultaneously an
interpretation and generalization bound (for the original DNN), connecting two
fronts which have historically been investigated separately. Unlike compression
techniques, our representation is. We motivate the utility of this perspective
by studying DNNs in simple, controlled settings, where we obtain superior
generalization bounds despite using only combinatorial information (e.g. no
margin information). We demonstrate how to "open the black box" on the MNIST
dataset. We show that the learned, internal, logical computations correspond to
semantically meaningful (unlabeled) categories that allow DNN descriptions in
plain English. We improve the generalization of an already trained network by
interpreting, diagnosing, and replacing components the logical circuit that is
the DNN.
