  We propose a scalable variational Bayes method for statistical inference for
a single or low-dimensional subset of the coordinates of a high-dimensional
parameter in sparse linear regression. Our approach relies on assigning a
mean-field approximation to the nuisance coordinates and carefully modelling
the conditional distribution of the target given the nuisance. This requires
only a preprocessing step and preserves the computational advantages of
mean-field variational Bayes, while ensuring accurate and reliable inference
for the target parameter, including for uncertainty quantification. We
investigate the numerical performance of our algorithm, showing that it
performs competitively with existing methods. We further establish accompanying
theoretical guarantees for estimation and uncertainty quantification in the
form of a Bernstein--von Mises theorem.
