  We introduce a methodology for designing and training deep neural networks
(DNN) that we call "Deep Regression Ensembles" (DRE). It bridges the gap
between DNN and two-layer neural networks trained with random feature
regression. Each layer of DRE has two components, randomly drawn input weights
and output weights trained myopically (as if the final output layer) using
linear ridge regression. Within a layer, each neuron uses a different subset of
inputs and a different ridge penalty, constituting an ensemble of random
feature ridge regressions. Our experiments show that a single DRE architecture
is at par with or exceeds state-of-the-art DNN in many data sets. Yet, because
DRE neural weights are either known in closed-form or randomly drawn, its
computational cost is orders of magnitude smaller than DNN.
