  In exciting new work, Bertsimas et al. (2016) showed that the classical best
subset selection problem in regression modeling can be formulated as a mixed
integer optimization (MIO) problem. Using recent advances in MIO algorithms,
they demonstrated that best subset selection can now be solved at much larger
problem sizes that what was thought possible in the statistics community. They
presented empirical comparisons of best subset selection with other popular
variable selection procedures, in particular, the lasso and forward stepwise
selection. Surprisingly (to us), their simulations suggested that best subset
selection consistently outperformed both methods in terms of prediction
accuracy. Here we present an expanded set of simulations to shed more light on
these comparisons.
  The summary is roughly as follows: (a) neither best subset selection nor the
lasso uniformly dominate the other, with best subset selection generally
performing better in high signal-to-noise (SNR) ratio regimes, and the lasso
better in low SNR regimes; (b) best subset selection and forward stepwise
perform quite similarly throughout; (c) the relaxed lasso (actually, a
simplified version of the original relaxed estimator defined in Meinshausen,
2007) is the overall winner, performing just about as well as the lasso in low
SNR scenarios, and as well as best subset selection in high SNR scenarios.
