  In modern data analysis, sparse model selection becomes inevitable once the
number of predictors variables is very high. It is well-known that model
selection procedures like the Lasso or Boosting tend to overfit on real data.
The celebrated Stability Selection overcomes these weaknesses by aggregating
models, based on subsamples of the training data, followed by choosing a stable
predictor set which is usually much sparser than the predictor sets from the
raw models. The standard Stability Selection is based on a global criterion,
namely the per-family error rate, while additionally requiring expert knowledge
to suitably configure the hyperparameters. Since model selection depends on the
loss function, i.e., predictor sets selected w.r.t. some particular loss
function differ from those selected w.r.t. some other loss function, we propose
a Stability Selection variant which respects the chosen loss function via an
additional validation step based on out-of-sample validation data, optionally
enhanced with an exhaustive search strategy. Our Stability Selection variants
are widely applicable and user-friendly. Moreover, our Stability Selection
variants can avoid the issue of severe underfitting which affects the original
Stability Selection for noisy high-dimensional data, so our priority is not to
avoid false positives at all costs but to result in a sparse stable model with
which one can make predictions. Experiments where we consider both regression
and binary classification and where we use Boosting as model selection
algorithm reveal a significant precision improvement compared to raw Boosting
models while not suffering from any of the mentioned issues of the original
Stability Selection.
