  Overparametrized neural networks tend to perfectly fit noisy training data
yet generalize well on test data. Inspired by this empirical observation,
recent work has sought to understand this phenomenon of benign overfitting or
harmless interpolation in the much simpler linear model. Previous theoretical
work critically assumes that either the data features are statistically
independent or the input data is high-dimensional; this precludes general
nonparametric settings with structured feature maps. In this paper, we present
a general and flexible framework for upper bounding regression and
classification risk in a reproducing kernel Hilbert space. A key contribution
is that our framework describes precise sufficient conditions on the data Gram
matrix under which harmless interpolation occurs. Our results recover prior
independent-features results (with a much simpler analysis), but they
furthermore show that harmless interpolation can occur in more general settings
such as features that are a bounded orthonormal system. Furthermore, our
results show an asymptotic separation between classification and regression
performance in a manner that was previously only shown for Gaussian features.
