  Objective: Provide guidance on sample size considerations for developing
predictive models by empirically establishing the adequate sample size, which
balances the competing objectives of improving model performance and reducing
model complexity as well as computational requirements.
  Materials and Methods: We empirically assess the effect of sample size on
prediction performance and model complexity by generating learning curves for
81 prediction problems (23 outcomes predicted in a depression cohort, 58
outcomes predicted in a hypertension cohort) in three large observational
health databases, requiring training of 17,248 prediction models. The adequate
sample size was defined as the sample size for which the performance of a model
equalled the maximum model performance minus a small threshold value.
  Results: The adequate sample size achieves a median reduction of the number
of observations of 9.5%, 37.3%, 58.5%, and 78.5% for the thresholds of 0.001,
0.005, 0.01, and 0.02, respectively. The median reduction of the number of
predictors in the models was 8.6%, 32.2%, 48.2%, and 68.3% for the thresholds
of 0.001, 0.005, 0.01, and 0.02, respectively.
  Discussion: Based on our results a conservative, yet significant, reduction
in sample size and model complexity can be estimated for future prediction
work. Though, if a researcher is willing to generate a learning curve a much
larger reduction of the model complexity may be possible as suggested by a
large outcome-dependent variability.
  Conclusion: Our results suggest that in most cases only a fraction of the
available data was sufficient to produce a model close to the performance of
one developed on the full data set, but with a substantially reduced model
complexity.
