  Score-based Generative Models (SGMs) have demonstrated exceptional synthesis
outcomes across various tasks. However, the current design landscape of the
forward diffusion process remains largely untapped and often relies on physical
heuristics or simplifying assumptions. Utilizing insights from the development
of scalable Bayesian posterior samplers, we present a complete recipe for
formulating forward processes in SGMs, ensuring convergence to the desired
target distribution. Our approach reveals that several existing SGMs can be
seen as specific manifestations of our framework. Building upon this method, we
introduce Phase Space Langevin Diffusion (PSLD), which relies on score-based
modeling within an augmented space enriched by auxiliary variables akin to
physical phase space. Empirical results exhibit the superior sample quality and
improved speed-quality trade-off of PSLD compared to various competing
approaches on established image synthesis benchmarks. Remarkably, PSLD achieves
sample quality akin to state-of-the-art SGMs (FID: 2.10 for unconditional
CIFAR-10 generation). Lastly, we demonstrate the applicability of PSLD in
conditional synthesis using pre-trained score networks, offering an appealing
alternative as an SGM backbone for future advancements. Code and model
checkpoints can be accessed at \url{https://github.com/mandt-lab/PSLD}.
