  Deep Neural Networks (DNNs) have obtained impressive performance across
tasks, however they still remain as black boxes, e.g., hard to theoretically
analyze. At the same time, Polynomial Networks (PNs) have emerged as an
alternative method with a promising performance and improved interpretability
but have yet to reach the performance of the powerful DNN baselines. In this
work, we aim to close this performance gap. We introduce a class of PNs, which
are able to reach the performance of ResNet across a range of six benchmarks.
We demonstrate that strong regularization is critical and conduct an extensive
study of the exact regularization schemes required to match performance. To
further motivate the regularization schemes, we introduce D-PolyNets that
achieve a higher-degree of expansion than previously proposed polynomial
networks. D-PolyNets are more parameter-efficient while achieving a similar
performance as other polynomial networks. We expect that our new models can
lead to an understanding of the role of elementwise activation functions (which
are no longer required for training PNs). The source code is available at
https://github.com/grigorisg9gr/regularized_polynomials.
