  Non-convex sampling is a key challenge in machine learning, central to
non-convex optimization in deep learning as well as to approximate
probabilistic inference. Despite its significance, theoretically there remain
many important challenges: Existing guarantees (1) typically only hold for the
averaged iterates rather than the more desirable last iterates, (2) lack
convergence metrics that capture the scales of the variables such as
Wasserstein distances, and (3) mainly apply to elementary schemes such as
stochastic gradient Langevin dynamics. In this paper, we develop a new
framework that lifts the above issues by harnessing several tools from the
theory of dynamical systems. Our key result is that, for a large class of
state-of-the-art sampling schemes, their last-iterate convergence in
Wasserstein distances can be reduced to the study of their continuous-time
counterparts, which is much better understood. Coupled with standard
assumptions of MCMC sampling, our theory immediately yields the last-iterate
Wasserstein convergence of many advanced sampling schemes such as proximal,
randomized mid-point, and Runge-Kutta integrators. Beyond existing methods, our
framework also motivates more efficient schemes that enjoy the same rigorous
guarantees.
