  Unbiased assessment of the predictivity of models learnt by supervised
machine-learning methods requires knowledge of the learned function over a
reserved test set (not used by the learning algorithm). The quality of the
assessment depends, naturally, on the properties of the test set and on the
error statistic used to estimate the prediction error. In this work we tackle
both issues, proposing a new predictivity criterion that carefully weights the
individual observed errors to obtain a global error estimate, and using
incremental experimental design methods to "optimally" select the test points
on which the criterion is computed. Several incremental constructions are
studied, including greedy-packing (coffee-house design), support points and
kernel herding techniques. Our results show that the incremental and weighted
versions of the latter two, based on Maximum Mean Discrepancy concepts, yield
superior performance. An industrial test case provided by the historical French
electricity supplier (EDF) illustrates the practical relevance of the
methodology, indicating that it is an efficient alternative to expensive
cross-validation techniques.
