  In this paper, we review state-of-the-art methods for feature selection in
statistics with an application-oriented eye. Indeed, sparsity is a valuable
property and the profusion of research on the topic might have provided little
guidance to practitioners. We demonstrate empirically how noise and correlation
impact both the accuracy - the number of correct features selected - and the
false detection - the number of incorrect features selected - for five methods:
the cardinality-constrained formulation, its Boolean relaxation, $\ell_1$
regularization and two methods with non-convex penalties. A cogent feature
selection method is expected to exhibit a two-fold convergence, namely the
accuracy and false detection rate should converge to $1$ and $0$ respectively,
as the sample size increases. As a result, proper method should recover all and
nothing but true features. Empirically, the integer optimization formulation
and its Boolean relaxation are the closest to exhibit this two properties
consistently in various regimes of noise and correlation. In addition, apart
from the discrete optimization approach which requires a substantial, yet often
affordable, computational time, all methods terminate in times comparable with
the \verb|glmnet| package for Lasso. We released code for methods that were not
publicly implemented. Jointly considered, accuracy, false detection and
computational time provide a comprehensive assessment of each feature selection
method and shed light on alternatives to the Lasso-regularization which are not
as popular in practice yet.
