  Lasso is a celebrated method for variable selection in linear models, but it
faces challenges when the variables are moderately or strongly correlated. This
motivates alternative approaches such as using a non-convex penalty, adding a
ridge regularization, or conducting a post-Lasso thresholding. In this paper,
we compare Lasso with 5 other methods: Elastic net, SCAD, forward selection,
thresholded Lasso, and forward backward selection. We measure their
performances theoretically by the expected Hamming error, assuming that the
regression coefficients are iid drawn from a two-point mixture and that the
Gram matrix is block-wise diagonal. By deriving the rates of convergence of
Hamming errors and the phase diagrams, we obtain useful conclusions about the
pros and cons of different methods.
