  Cross-validation (CV) is widely used for tuning a model with respect to
user-selected parameters and for selecting a "best" model. For example, the
method of $k$-nearest neighbors requires the user to choose $k$, the number of
neighbors, and a neural network has several tuning parameters controlling the
network complexity. Once such parameters are optimized for a particular data
set, the next step is often to compare the various optimized models and choose
the method with the best predictive performance. Both tuning and model
selection boil down to comparing models, either across different values of the
tuning parameters or across different classes of statistical models and/or sets
of explanatory variables. For multiple large sets of data, like the PubChem
drug discovery cheminformatics data which motivated this work, reliable CV
comparisons are computationally demanding, or even infeasible. In this paper we
develop an efficient sequential methodology for model comparison based on CV.
It also takes into account the randomness in CV. The number of models is
reduced via an adaptive, multiplicity-adjusted sequential algorithm, where poor
performers are quickly eliminated. By exploiting matching of individual
observations, it is sometimes even possible to establish the statistically
significant inferiority of some models with just one execution of CV.
