  We argue that comprehensive out-of-sample (OOS) evaluation using statistical
decision theory (SDT) should replace the current practice of K-fold and Common
Task Framework validation in machine learning (ML) research. SDT provides a
formal framework for performing comprehensive OOS evaluation across all
possible (1) training samples, (2) populations that may generate training data,
and (3) populations of prediction interest. Regarding feature (3), we emphasize
that SDT requires the practitioner to directly confront the possibility that
the future may not look like the past and to account for a possible need to
extrapolate from one population to another when building a predictive
algorithm. SDT is simple in abstraction, but it is often computationally
demanding to implement. We discuss progress in tractable implementation of SDT
when prediction accuracy is measured by mean square error or by
misclassification rate. We summarize research studying settings in which the
training data will be generated from a subpopulation of the population of
prediction interest. We consider conditional prediction with alternative
restrictions on the state space of possible populations that may generate
training data. We present an illustrative application of the methodology to the
problem of predicting patient illness to inform clinical decision making. We
conclude by calling on ML researchers to join with econometricians and
statisticians in expanding the domain within which implementation of SDT is
tractable.
