  Online nonparametric estimators are gaining popularity due to their efficient
computation and competitive generalization abilities. An important example
includes variants of stochastic gradient descent. These algorithms often take
one sample point at a time and instantly update the parameter estimate of
interest. In this work we consider model selection and hyperparameter tuning
for such online algorithms. We propose a weighted rolling-validation procedure,
an online variant of leave-one-out cross-validation, that costs minimal extra
computation for many typical stochastic gradient descent estimators. Similar to
batch cross-validation, it can boost base estimators to achieve a better,
adaptive convergence rate. Our theoretical analysis is straightforward, relying
mainly on some general statistical stability assumptions. The simulation study
underscores the significance of diverging weights in rolling validation in
practice and demonstrates its sensitivity even when there is only a slim
difference between candidate estimators.
