  Denoising Diffusion Probabilistic Models (DDPMs) have achieved impressive
performance on various generation tasks. By modeling the reverse process of
gradually diffusing the data distribution into a Gaussian distribution,
generating a sample in DDPMs can be regarded as iteratively denoising a
randomly sampled Gaussian noise. However, in practice DDPMs often need hundreds
even thousands of denoising steps to obtain a high-quality sample from the
Gaussian noise, leading to extremely low inference efficiency. In this work, we
propose a principled acceleration strategy, referred to as Early-Stopped DDPM
(ES-DDPM), for DDPMs. The key idea is to stop the diffusion process early where
only the few initial diffusing steps are considered and the reverse denoising
process starts from a non-Gaussian distribution. By further adopting a powerful
pre-trained generative model, such as GAN and VAE, in ES-DDPM, sampling from
the target non-Gaussian distribution can be efficiently achieved by diffusing
samples obtained from the pre-trained generative model. In this way, the number
of required denoising steps is significantly reduced. In the meantime, the
sample quality of ES-DDPM also improves substantially, outperforming both the
vanilla DDPM and the adopted pre-trained generative model. On extensive
experiments across CIFAR-10, CelebA, ImageNet, LSUN-Bedroom and LSUN-Cat,
ES-DDPM obtains promising acceleration effect and performance improvement over
representative baseline methods. Moreover, ES-DDPM also demonstrates several
attractive properties, including being orthogonal to existing acceleration
methods, as well as simultaneously enabling both global semantic and local
pixel-level control in image generation.
