  Fitting models with high predictive accuracy that include all relevant but no
irrelevant or redundant features is a challenging task on data sets with
similar (e.g. highly correlated) features. We propose the approach of tuning
the hyperparameters of a predictive model in a multi-criteria fashion with
respect to predictive accuracy and feature selection stability. We evaluate
this approach based on both simulated and real data sets and we compare it to
the standard approach of single-criteria tuning of the hyperparameters as well
as to the state-of-the-art technique "stability selection". We conclude that
our approach achieves the same or better predictive performance compared to the
two established approaches. Considering the stability during tuning does not
decrease the predictive accuracy of the resulting models. Our approach succeeds
at selecting the relevant features while avoiding irrelevant or redundant
features. The single-criteria approach fails at avoiding irrelevant or
redundant features and the stability selection approach fails at selecting
enough relevant features for achieving acceptable predictive accuracy. For our
approach, for data sets with many similar features, the feature selection
stability must be evaluated with an adjusted stability measure, that is, a
measure that considers similarities between features. For data sets with only
few similar features, an unadjusted stability measure suffices and is faster to
compute.
