  Consider the problem of high dimensional variable selection for the Gaussian
linear model when the unknown error variance is also of interest. In this
paper, we show that the use of conjugate shrinkage priors for Bayesian variable
selection can have detrimental consequences for such variance estimation. Such
priors are often motivated by the invariance argument of Jeffreys (1961).
Revisiting this work, however, we highlight a caveat that Jeffreys himself
noticed; namely that biased estimators can result from inducing dependence
between parameters a priori. In a similar way, we show that conjugate priors
for linear regression, which induce prior dependence, can lead to such
underestimation in the Bayesian high-dimensional regression setting. Following
Jeffreys, we recommend as a remedy to treat regression coefficients and the
error variance as independent a priori. Using such an independence prior
framework, we extend the Spike-and-Slab Lasso of Rockova and George (2018) to
the unknown variance case. This extended procedure outperforms both the fixed
variance approach and alternative penalized likelihood methods on simulated
data. On the protein activity dataset of Clyde and Parmigiani (1998), the
Spike-and-Slab Lasso with unknown variance achieves lower cross-validation
error than alternative penalized likelihood methods, demonstrating the gains in
predictive accuracy afforded by simultaneous error variance estimation.
