  Series and polynomial regression are able to approximate the same function
classes as neural networks. However, these methods are rarely used in practice,
although they offer more interpretability than neural networks. In this paper,
we show that a potential reason for this is the slow convergence rate of
polynomial regression estimators and propose the use of bagged polynomial
regression (BPR) as an attractive alternative to neural networks.
Theoretically, we derive new finite sample and asymptotic $L^2$ convergence
rates for series estimators. We show that the rates can be improved in smooth
settings by splitting the feature space and generating polynomial features
separately for each partition. Empirically, we show that our proposed
estimator, the BPR, can perform as well as more complex models with more
parameters. Our estimator also performs close to state-of-the-art prediction
methods in the benchmark MNIST handwritten digit dataset.
