  Recent theoretical studies illustrated that kernel ridgeless regression can
guarantee good generalization ability without an explicit regularization. In
this paper, we investigate the statistical properties of ridgeless regression
with random features and stochastic gradient descent. We explore the effect of
factors in the stochastic gradient and random features, respectively.
Specifically, random features error exhibits the double-descent curve.
Motivated by the theoretical findings, we propose a tunable kernel algorithm
that optimizes the spectral density of kernel during training. Our work bridges
the interpolation theory and practical algorithm.
