  Variable selection in linear regression settings is a much discussed problem.
Best subset selection (BSS) is often considered the intuitive 'gold standard',
with its use being restricted only by its NP-hard nature. Alternatives such as
the least absolute shrinkage and selection operator (Lasso) or the elastic net
(Enet) have become methods of choice in high-dimensional settings. A recent
proposal represents BSS as a mixed integer optimization problem so that much
larger problems have become feasible in reasonable computation time. We present
an extensive neutral comparison assessing the variable selection performance,
in linear regressions, of BSS compared to forward stepwise selection (FSS),
Lasso and Enet. The simulation study considers a wide range of settings that
are challenging with regard to dimensionality (with respect to the number of
observations and variables), signal-to-noise ratios and correlations between
predictors. As main measure of performance, we used the best possible F1-score
for each method to ensure a fair comparison irrespective of any criterion for
choosing the tuning parameters, and results were confirmed by alternative
performance measures. Somewhat surprisingly, it was only in settings where the
signal-to-noise ratio was high and the variables were (nearly) uncorrelated
that BSS reliably outperformed the other methods, even in low-dimensional
settings. Further, the FSS's performance was nearly identical to BSS. Our
results shed new light on the usual presumption of BSS being, in principle, the
best choice for variable selection. Especially for correlated variables,
alternatives like Enet are faster and appear to perform better in practical
settings.
