  Scalable and accurate identification of specific clinical outcomes has been
enabled by machine-learning applied to electronic medical record (EMR) systems.
The development of classification models requires the collection of a complete
labeled data set, where true clinical outcomes are obtained by human expert
manual review. For example, the development of natural language processing
algorithms requires the abstraction of clinical text data to obtain outcome
information necessary for training models. However, if the outcome is rare then
simple random sampling results in very few cases and insufficient information
to develop accurate classifiers. Since large scale detailed abstraction is
often expensive, time-consuming, and not feasible, more efficient strategies
are needed. Under such resource constrained settings, we propose a class of
enrichment sampling designs, where selection for abstraction is stratified by
auxiliary variables related to the true outcome of interest. Stratified
sampling on highly specific variables results in targeted samples that are more
enriched with cases, which we show translates to increased model discrimination
and better statistical learning performance. We provide mathematical details,
and simulation evidence that links sampling designs to their resulting
prediction model performance. We discuss the impact of our proposed sampling on
both model training and validation. Finally, we illustrate the proposed designs
for outcome label collection and subsequent machine-learning, using radiology
report text data from the Lumbar Imaging with Reporting of Epidemiology (LIRE)
study.
