  Today, it is more important than ever before for users to have trust in the
models they use. As Machine Learning models fall under increased regulatory
scrutiny and begin to see more applications in high-stakes situations, it
becomes critical to explain our models. Piecewise Linear Neural Networks (PLNN)
with the ReLU activation function have quickly become extremely popular models
due to many appealing properties; however, they still present many challenges
in the areas of robustness and interpretation. To this end, we introduce novel
methodology toward simplification and increased interpretability of Piecewise
Linear Neural Networks for classification tasks. Our methods include the use of
a trained, deep network to produce a well-performing, single-hidden-layer
network without further stochastic training, in addition to an algorithm to
reduce flat networks to a smaller, more interpretable size with minimal loss in
performance. On these methods, we conduct preliminary studies of model
performance, as well as a case study on Wells Fargo's Home Lending dataset,
together with visual model interpretation.
