  The success of over-parameterized neural networks trained to near-zero
training error has caused great interest in the phenomenon of benign
overfitting, where estimators are statistically consistent even though they
interpolate noisy training data. While benign overfitting in fixed dimension
has been established for some learning methods, current literature suggests
that for regression with typical kernel methods and wide neural networks,
benign overfitting requires a high-dimensional setting where the dimension
grows with the sample size. In this paper, we show that the smoothness of the
estimators, and not the dimension, is the key: benign overfitting is possible
if and only if the estimator's derivatives are large enough. We generalize
existing inconsistency results to non-interpolating models and more kernels to
show that benign overfitting with moderate derivatives is impossible in fixed
dimension. Conversely, we show that rate-optimal benign overfitting is possible
for regression with a sequence of spiky-smooth kernels with large derivatives.
Using neural tangent kernels, we translate our results to wide neural networks.
We prove that while infinite-width networks do not overfit benignly with the
ReLU activation, this can be fixed by adding small high-frequency fluctuations
to the activation function. Our experiments verify that such neural networks,
while overfitting, can indeed generalize well even on low-dimensional data
sets.
