  Feature selection, identifying a subset of variables that are relevant for
predicting a response, is an important and challenging component of many
methods in statistics and machine learning. Feature selection is especially
difficult and computationally intensive when the number of variables approaches
or exceeds the number of samples, as is often the case for many genomic
datasets. Here, we introduce a new approach -- the Bayesian Ising Approximation
(BIA) -- to rapidly calculate posterior probabilities for feature relevance in
L2 penalized linear regression. In the regime where the regression problem is
strongly regularized by the prior, we show that computing the marginal
posterior probabilities for features is equivalent to computing the
magnetizations of an Ising model. Using a mean field approximation, we show it
is possible to rapidly compute the feature selection path described by the
posterior probabilities as a function of the L2 penalty. We present simulations
and analytical results illustrating the accuracy of the BIA on some simple
regression problems. Finally, we demonstrate the applicability of the BIA to
high dimensional regression by analyzing a gene expression dataset with nearly
30,000 features.
