  We study the Bayesian approach to variable selection in the context of linear
regression. Motivated by a recent work by Rockova and George (2014), we propose
an EM algorithm that returns the MAP estimate of the set of relevant variables.
Due to its particular updating scheme, our algorithm can be implemented
efficiently without inverting a large matrix in each iteration and therefore
can scale up with big data. We also show that the MAP estimate returned by our
EM algorithm achieves variable selection consistency even when $p$ diverges
with $n$. In practice, our algorithm could get stuck with local modes, a common
problem with EM algorithms. To address this issue, we propose an ensemble EM
algorithm, in which we repeatedly apply the EM algorithm on a subset of the
samples with a subset of the covariates, and then aggregate the variable
selection results across those bootstrap replicates. Empirical studies have
demonstrated the superior performance of the ensemble EM algorithm.
