  We study empirical Bayes estimation in high-dimensional linear regression. To
facilitate computationally efficient estimation of the underlying prior, we
adopt a variational empirical Bayes approach, introduced originally in
Carbonetto and Stephens (2012) and Kim et al. (2022). We establish asymptotic
consistency of the nonparametric maximum likelihood estimator (NPMLE) and its
(computable) naive mean field variational surrogate under mild assumptions on
the design and the prior. Assuming, in addition, that the naive mean field
approximation has a dominant optimizer, we develop a computationally efficient
approximation to the oracle posterior distribution, and establish its accuracy
under the 1-Wasserstein metric. This enables computationally feasible Bayesian
inference; e.g., construction of posterior credible intervals with an average
coverage guarantee, Bayes optimal estimation for the regression coefficients,
estimation of the proportion of non-nulls, etc. Our analysis covers both
deterministic and random designs, and accommodates correlations among the
features. To the best of our knowledge, this provides the first rigorous
nonparametric empirical Bayes method in a high-dimensional regression setting
without sparsity.
