  Bayesian shrinkage methods have generated a lot of recent interest as tools
for high-dimensional regression and model selection. These methods naturally
facilitate tractable uncertainty quantification and incorporation of prior
information. A common feature of these models, including the Bayesian lasso,
global-local shrinkage priors, and spike-and-slab priors is that the
corresponding priors on the regression coefficients can be expressed as scale
mixture of normals. While the three-step Gibbs sampler used to sample from the
often intractable associated posterior density has been shown to be
geometrically ergodic for several of these models (Khare and Hobert, 2013; Pal
and Khare, 2014), it has been demonstrated recently that convergence of this
sampler can still be quite slow in modern high-dimensional settings despite
this apparent theoretical safeguard. We propose a new method to draw from the
same posterior via a tractable two-step blocked Gibbs sampler. We demonstrate
that our proposed two-step blocked sampler exhibits vastly superior convergence
behavior compared to the original three- step sampler in high-dimensional
regimes on both real and simulated data. We also provide a detailed theoretical
underpinning to the new method in the context of the Bayesian lasso. First, we
derive explicit upper bounds for the (geometric) rate of convergence.
Furthermore, we demonstrate theoretically that while the original Bayesian
lasso chain is not Hilbert-Schmidt, the proposed chain is trace class (and
hence Hilbert-Schmidt). The trace class property has useful theoretical and
practical implications. It implies that the corresponding Markov operator is
compact, and its eigenvalues are summable. It also facilitates a rigorous
comparison of the two-step blocked chain with "sandwich" algorithms which aim
to improve performance of the two-step chain by inserting an inexpensive extra
step.
