  Prediction models for clinical outcomes may be developed using a source
dataset and additionally applied to new settings. Towards model external
validation and model updating in the new setting, one procedure is model
modification learning that involves the dual goals of recalibrating overall
predictions as well as revising individual feature effects. Modification
learning generally requires the collection of an adequate sample of true
outcome labels from the new setting, which is frequently an expensive and
time-consuming process, as it involves abstraction by human clinical experts.
To reduce the abstraction burden for such new data collection, we propose a
class of designs based on original model scores and their associated outcome
predictions. We provide mathematical justification that the general predictive
score sampling class results in valid samples for analysis. Then, we focus
attention specifically on a stratified sampling procedure that we call
predictive case control (PCC) sampling, which allows the dual modification
learning goals to be achieved at a smaller sample size compared to simple
random sampling (SRS). PCC sampling intentionally over-represents subjects with
informative scores, where we suggest using the D-optimality and Binary Entropy
information functions to summarize sample information. For design evaluation
within the PCC class, we provide a computational framework to estimate and
visualize empirical response surfaces of the proposed information functions. We
demonstrate the benefit of using PCC designs for modification learning,
relative to SRS, through Monte Carlo simulation. Finally, using radiology
report data from the Lumbar Imaging with Reporting of Epidemiology (LIRE)
study, we illustrate the application of PCC for new outcome label abstraction
and subsequent modification learning across imaging modalities.
