  Applied statisticians use sequential regression procedures to produce a
ranking of explanatory variables and, in settings of low correlations between
variables and strong true effect sizes, expect that variables at the very top
of this ranking are truly relevant to the response. In a regime of certain
sparsity levels, however, three examples of sequential procedures--forward
stepwise, the lasso, and least angle regression--are shown to include the first
spurious variable unexpectedly early. We derive a rigorous, sharp prediction of
the rank of the first spurious variable for these three procedures,
demonstrating that the first spurious variable occurs earlier and earlier as
the regression coefficients become denser. This counterintuitive phenomenon
persists for statistically independent Gaussian random designs and an
arbitrarily large magnitude of the true effects. We gain a better understanding
of the phenomenon by identifying the underlying cause and then leverage the
insights to introduce a simple visualization tool termed the double-ranking
diagram to improve on sequential methods. As a byproduct of these findings, we
obtain the first provable result certifying the exact equivalence between the
lasso and least angle regression in the early stages of solution paths beyond
orthogonal designs. This equivalence can seamlessly carry over many important
model selection results concerning the lasso to least angle regression.
