  Stability selection (Meinshausen and Buhlmann, 2010) makes any feature
selection method more stable by returning only those features that are
consistently selected across many subsamples. We prove (in what is, to our
knowledge, the first result of its kind) that for data containing highly
correlated proxies for an important latent variable, the lasso typically
selects one proxy, yet stability selection with the lasso can fail to select
any proxy, leading to worse predictive performance than the lasso alone.
  We introduce cluster stability selection, which exploits the practitioner's
knowledge that highly correlated clusters exist in the data, resulting in
better feature rankings than stability selection in this setting. We consider
several feature-combination approaches, including taking a weighted average of
the features in each important cluster where weights are determined by the
frequency with which cluster members are selected, which we show leads to
better predictive models than previous proposals.
  We present generalizations of theoretical guarantees from Meinshausen and
Buhlmann (2010) and Shah and Samworth (2012) to show that cluster stability
selection retains the same guarantees. In summary, cluster stability selection
enjoys the best of both worlds, yielding a sparse selected set that is both
stable and has good predictive performance.
