  Penalized likelihood approaches are widely used for high-dimensional
regression. Although many methods have been proposed and the associated theory
is now well-developed, the relative efficacy of different approaches in
finite-sample settings, as encountered in practice, remains incompletely
understood. There is therefore a need for empirical investigations in this area
that can offer practical insight and guidance to users. In this paper we
present a large-scale comparison of penalized regression methods. We
distinguish between three related goals: prediction, variable selection and
variable ranking. Our results span more than 2,300 data-generating scenarios,
including both synthetic and semi-synthetic data (real covariates and simulated
responses), allowing us to systematically consider the influence of various
factors (sample size, dimensionality, sparsity, signal strength and
multicollinearity). We consider several widely-used approaches (Lasso, Adaptive
Lasso, Elastic Net, Ridge Regression, SCAD, the Dantzig Selector and Stability
Selection). We find considerable variation in performance between methods. Our
results support a `no panacea' view, with no unambiguous winner across all
scenarios or goals, even in this restricted setting where all data align well
with the assumptions underlying the methods. The study allows us to make some
recommendations as to which approaches may be most (or least) suitable given
the goal and some data characteristics. Our empirical results complement
existing theory and provide a resource to compare methods across a range of
scenarios and metrics.
