  As the discipline has evolved, research in machine learning has been focused
more and more on creating more powerful neural networks, without regard for the
interpretability of these networks. Such "black-box models" yield
state-of-the-art results, but we cannot understand why they make a particular
decision or prediction. Sometimes this is acceptable, but often it is not. We
propose a novel architecture, Regression Networks, which combines the power of
neural networks with the understandability of regression analysis. While some
methods for combining these exist in the literature, our architecture
generalizes these approaches by taking interactions into account, offering the
power of a dense neural network without forsaking interpretability. We
demonstrate that the models exceed the state-of-the-art performance of
interpretable models on several benchmark datasets, matching the power of a
dense neural network. Finally, we discuss how these techniques can be
generalized to other neural architectures, such as convolutional and recurrent
neural networks.
