  Current high-throughput technologies provide a large amount of variables to
describe a phenomenon. Only a few variables are generally sufficient to answer
the question. Identify them in a high-dimensional Gaussian linear regression
model is the one of the most-used statistical methods. In this article, we
describe step-by-step the variable selection procedures built upon
regularization paths. Regularization paths are obtained by combining a
regularization function and an algorithm. Then, they are combined either with a
model selection procedure using penalty functions or with a sampling strategy
to obtain the final selected variables. We perform a comparison study by
considering three simulation settings with various dependency structures on
variables. %from the most classical to a most realistic one. In all the
settings, we evaluate (i) the ability to discriminate between the active
variables and the non-active variables along the regularization path
(pROC-AUC), (ii) the prediction performance of the selected variable subset
(MSE) and (iii) the relevance of the selected variables (recall, specificity,
FDR). From the results, we provide recommendations on strategies to be favored
depending on the characteristics of the problem at hand. We obtain that the
regularization function Elastic-net provides most of the time better results
than the $\ell_1$ one and the lars algorithm has to be privileged as the GD
one. ESCV provides the best prediction performances. Bolasso and the knockoffs
method are judicious choices to limit the selection of non-active variables
while ensuring selection of enough active variables. Conversely, the
data-driven penalties considered in this review are not to be favored. As for
Tigress and LinSelect, they are conservative methods.
