  Bayesian shrinkage methods have generated a lot of recent interest as tools
for high-dimensional regression and model selection. These methods naturally
facilitate tractable uncertainty quantification and incorporation of prior
information. This benefit has led to extensive use of the Bayesian shrinkage
methods across diverse applications. A common feature of these models is that
the corresponding priors on the regression coefficients can be expressed as
scale mixture of normals. While the three-step Gibbs sampler used to sample
from the often intractable associated posterior density has been shown to be
geometrically ergodic for several of these models, it has been demonstrated
recently that convergence of this sampler can still be quite slow in modern
high-dimensional settings despite this apparent theoretical safeguard. We
propose a new method to draw from the same posterior via a tractable two-step
blocked Gibbs sampler. We demonstrate that our proposed two-step blocked
sampler exhibits vastly superior convergence behavior compared to the original
three-step sampler in high-dimensional regimes on both real and simulated data.
We also provide a detailed theoretical underpinning to the new method in the
context of the Bayesian lasso. First, we prove that the proposed two-step
sampler is geometrically ergodic, and derive explicit upper bounds for the
(geometric) rate of convergence. Furthermore, we demonstrate theoretically that
while the original Bayesian lasso chain is not Hilbert-Schmidt, the proposed
chain is trace class (and hence Hilbert-Schmidt). The trace class property
implies that the corresponding Markov operator is compact, and its (countably
many) eigenvalues are summable. It also facilitates a rigorous comparison of
the two-step blocked chain with "sandwich" algorithms which aim to improve
performance of the two-step chain by inserting an inexpensive extra step.
