  Feature selection problems have been extensively studied for linear
estimation, for instance, Lasso, but less emphasis has been placed on feature
selection for non-linear functions. In this study, we propose a method for
feature selection in high-dimensional non-linear function estimation problems.
The new procedure is based on minimizing the $\ell_0$ norm of the vector of
indicator variables that represent if a feature is selected or not. Our
approach relies on the continuous relaxation of Bernoulli distributions, which
allows our model to learn the parameters of the approximate Bernoulli
distributions via gradient descent. This general framework simultaneously
minimizes a loss function while selecting relevant features. Furthermore, we
provide an information-theoretic justification of incorporating Bernoulli
distribution into our approach and demonstrate the potential of the approach on
synthetic and real-life applications.
