  Overparameterized neural networks (NNs) are observed to generalize well even
when trained to perfectly fit noisy data. This phenomenon motivated a large
body of work on "benign overfitting", where interpolating predictors achieve
near-optimal performance. Recently, it was conjectured and empirically observed
that the behavior of NNs is often better described as "tempered overfitting",
where the performance is non-optimal yet also non-trivial, and degrades as a
function of the noise level. However, a theoretical justification of this claim
for non-linear NNs has been lacking so far. In this work, we provide several
results that aim at bridging these complementing views. We study a simple
classification setting with 2-layer ReLU NNs, and prove that under various
assumptions, the type of overfitting transitions from tempered in the extreme
case of one-dimensional data, to benign in high dimensions. Thus, we show that
the input dimension has a crucial role on the type of overfitting in this
setting, which we also validate empirically for intermediate dimensions.
Overall, our results shed light on the intricate connections between the
dimension, sample size, architecture and training algorithm on the one hand,
and the type of resulting overfitting on the other hand.
