  When developing risk prediction models, shrinkage methods are recommended,
especially when the sample size is limited. Several earlier studies have shown
that the shrinkage of model coefficients can reduce overfitting of the
prediction model and subsequently result in better predictive performance on
average. In this simulation study, we aimed to investigate the variability of
regression shrinkage on predictive performance for a binary outcome, with focus
on the calibration slope. The slope indicates whether risk predictions are too
extreme (slope < 1) or not extreme enough (slope > 1). We investigated the
following shrinkage methods in comparison to standard maximum likelihood
estimation: uniform shrinkage (likelihood-based and bootstrap-based), ridge
regression, penalized maximum likelihood, LASSO regression, adaptive LASSO,
non-negative garrote, and Firth's correction. There were three main findings.
First, shrinkage improved calibration slopes on average. Second, the
between-sample variability of calibration slopes was often increased relative
to maximum likelihood. Among the shrinkage methods, the bootstrap-based uniform
shrinkage worked well overall. In contrast to other shrinkage approaches,
Firth's correction had only a small shrinkage effect but did so with low
variability. Third, the correlation between the estimated shrinkage and the
optimal shrinkage to remove overfitting was typically negative. Hence, although
shrinkage improved predictions on average, it often worked poorly in individual
datasets, in particular when shrinkage was most needed. The observed
variability of shrinkage methods implies that these methods do not solve
problems associated with small sample size or low number of events per
variable.
