  Recent empirical and theoretical studies have shown that many learning
algorithms -- from linear regression to neural networks -- can have test
performance that is non-monotonic in quantities such the sample size and model
size. This striking phenomenon, often referred to as "double descent", has
raised questions of if we need to re-think our current understanding of
generalization. In this work, we study whether the double-descent phenomenon
can be avoided by using optimal regularization. Theoretically, we prove that
for certain linear regression models with isotropic data distribution,
optimally-tuned $\ell_2$ regularization achieves monotonic test performance as
we grow either the sample size or the model size. We also demonstrate
empirically that optimally-tuned $\ell_2$ regularization can mitigate double
descent for more general models, including neural networks. Our results suggest
that it may also be informative to study the test risk scalings of various
algorithms in the context of appropriately tuned regularization.
