  Databases of electronic health records (EHRs) are increasingly used to inform
clinical decisions. Machine learning methods can find patterns in EHRs that are
predictive of future adverse outcomes. However, statistical models may be built
upon patterns of health-seeking behavior that vary across patient
subpopulations, leading to poor predictive performance when training on one
patient population and predicting on another. This note proposes two tests to
better measure and understand model generalization. We use these tests to
compare models derived from two data sources: (i) historical medical records,
and (ii) electrocardiogram (EKG) waveforms. In a predictive task, we show that
EKG-based models can be more stable than EHR-based models across different
patient populations.
