  Cross-validation assesses the predictive ability of a model, allowing one to
rank models accordingly. Although the nonparametric bootstrap is almost always
used to assess the variability of a parameter, it can be used as the basis for
cross-validation if one keeps track of which items were not selected in a given
bootstrap iteration. The items which were selected constitute the training data
and the omitted items constitute the testing data. This bootstrap
cross-validation (BS-CV) allows model selection to be made on the basis of
predictive ability by comparing the median values of ensembles of summary
statistics of testing data. BS-CV is herein demonstrated using several summary
statistics, including a new one termed the simple metric for prediction quality
(SMPQ), and using the warfarin data included in the Monolix distribution with
13 pharmacokinetics (PK) models and 12 pharmacodynamics (PD) models. Of note
the two best PK models by AIC had the worst predictive ability, underscoring
the danger of using single realizations of a random variable (such as AIC) as
the basis for model selection. Using these data BS-CV was able to discriminate
between similar indirect response models (inhibition of input versus
stimulation of output). This could be useful in situations in which the
mechanism of action is unknown (unlike warfarin).
