  Markov chain Monte Carlo (MCMC) lies at the core of modern Bayesian
methodology, much of which would be impossible without it. Thus, the
convergence properties of MCMCs have received significant attention, and in
particular, proving (geometric) ergodicity is of critical interest. Trust in
the ability of MCMCs to sample from modern-day high-dimensional posteriors,
however, has been limited by a widespread perception that these chains
typically experience serious convergence problems. In this paper, we first
demonstrate that contemporary methods for obtaining convergence rates have
serious limitations when the dimension grows. We then propose a framework for
rigorously establishing the convergence behavior of commonly used
high-dimensional MCMCs. In particular, we demonstrate theoretically the precise
nature and severity of the convergence problems of popular MCMCs when
implemented in high dimensions, including phase transitions in the convergence
rates in various $n$ and $p$ regimes, and a universality result across an
entire spectrum of models. We also show that convergence problems effectively
eliminate the apparent safeguard of geometric ergodicity. We then demonstrate
theoretical principles by which MCMCs can be constructed and analyzed to yield
bounded geometric convergence rates even as the dimension $p$ grows without
bound. Additionally, we propose a diagnostic tool for establishing convergence.
