  In this paper, we study the challenge of feature selection based on a
relatively small collection of sample pairs $\{(x_i, y_i)\}_{1 \leq i \leq m}$.
The observations $y_i \in \mathbb{R}$ are thereby supposed to follow a noisy
single-index model, depending on a certain set of signal variables. A major
difficulty is that these variables usually cannot be observed directly, but
rather arise as hidden factors in the actual data vectors $x_i \in
\mathbb{R}^d$ (feature variables). We will prove that a successful variable
selection is still possible in this setup, even when the applied estimator does
not have any knowledge of the underlying model parameters and only takes the
'raw' samples $\{(x_i, y_i)\}_{1 \leq i \leq m}$ as input. The model
assumptions of our results will be fairly general, allowing for non-linear
observations, arbitrary convex signal structures as well as strictly convex
loss functions. This is particularly appealing for practical purposes, since in
many applications, already standard methods, e.g., the Lasso or logistic
regression, yield surprisingly good outcomes. Apart from a general discussion
of the practical scope of our theoretical findings, we will also derive a
rigorous guarantee for a specific real-world problem, namely sparse feature
extraction from (proteomics-based) mass spectrometry data.
