  Leading methods for support recovery in high-dimensional regression, such as
Lasso, have been well-studied and their limitations in the context of
correlated design have been characterized with precise incoherence conditions.
In this work, we present a similar treatment of selection consistency for
marginal regression (MR), a computationally efficient family of methods with
connections to decision trees. Selection based on marginal regression is also
referred to as covariate screening or independence screening and is a popular
approach in applied work, especially in ultra high-dimensional settings. We
identify the underlying factors---which we denote as \emph{MR
incoherence}---affecting MR's support recovery performance. Our near complete
characterization provides a much more nuanced and optimistic view of MR in
comparison to previous works. To ground our results, we provide a broad
taxonomy of results for leading feature selection methods, relating the
behavior of Lasso, OMP, SIS, and MR. We also lay the foundation for interesting
generalizations of our analysis, e.g., to non-linear feature selection methods
and to more general regression frameworks such as a general additive models.
