  Understanding how overparameterized neural networks generalize despite
perfect interpolation of noisy training data is a fundamental question.
Mallinar et. al. 2022 noted that neural networks seem to often exhibit
``tempered overfitting'', wherein the population risk does not converge to the
Bayes optimal error, but neither does it approach infinity, yielding
non-trivial generalization. However, this has not been studied rigorously. We
provide the first rigorous analysis of the overfitting behavior of regression
with minimum norm ($\ell_2$ of weights), focusing on univariate two-layer ReLU
networks. We show overfitting is tempered (with high probability) when measured
with respect to the $L_1$ loss, but also show that the situation is more
complex than suggested by Mallinar et. al., and overfitting is catastrophic
with respect to the $L_2$ loss, or when taking an expectation over the training
set.
