  ``Benign overfitting'', the ability of certain algorithms to interpolate
noisy training data and yet perform well out-of-sample, has been a topic of
considerable recent interest. We show, using a fixed design setup, that an
important class of predictors, kernel machines with translation-invariant
kernels, does not exhibit benign overfitting in fixed dimensions. In
particular, the estimated predictor does not converge to the ground truth with
increasing sample size, for any non-zero regression function and any (even
adaptive) bandwidth selection. To prove these results, we give exact
expressions for the generalization error, and its decomposition in terms of an
approximation error and an estimation error that elicits a trade-off based on
the selection of the kernel bandwidth. Our results apply to commonly used
translation-invariant kernels such as Gaussian, Laplace, and Cauchy.
