  Score-based generative models (SGMs) have recently emerged as a promising
class of generative models. However, a fundamental limitation is that their
sampling process is slow due to a need for many (\eg, $2000$) iterations of
sequential computations. An intuitive acceleration method is to reduce the
sampling iterations which however causes severe performance degradation. We
assault this problem to the ill-conditioned issues of the Langevin dynamics and
reverse diffusion in the sampling process. Under this insight, we propose a
model-agnostic {\bf\em preconditioned diffusion sampling} (PDS) method that
leverages matrix preconditioning to alleviate the aforementioned problem. PDS
alters the sampling process of a vanilla SGM at marginal extra computation
cost, and without model retraining. Theoretically, we prove that PDS preserves
the output distribution of the SGM, no risk of inducing systematical bias to
the original sampling process. We further theoretically reveal a relation
between the parameter of PDS and the sampling iterations,easing the parameter
estimation under varying sampling iterations. Extensive experiments on various
image datasets with a variety of resolutions and diversity validate that our
PDS consistently accelerates off-the-shelf SGMs whilst maintaining the
synthesis quality. In particular, PDS can accelerate by up to $29\times$ on
more challenging high resolution (1024$\times$1024) image generation. Compared
with the latest generative models (\eg, CLD-SGM, DDIM, and Analytic-DDIM), PDS
can achieve the best sampling quality on CIFAR-10 at a FID score of 1.99. Our
code is made publicly available to foster any further research
https://github.com/fudan-zvg/PDS.
