  A diffusion model, which is formulated to produce an image using thousands of
denoising steps, usually suffers from a slow inference speed. Existing
acceleration algorithms simplify the sampling by skipping most steps yet
exhibit considerable performance degradation. By viewing the generation of
diffusion models as a discretized integrating process, we argue that the
quality drop is partly caused by applying an inaccurate integral direction to a
timestep interval. To rectify this issue, we propose a timestep aligner that
helps find a more accurate integral direction for a particular interval at the
minimum cost. Specifically, at each denoising step, we replace the original
parameterization by conditioning the network on a new timestep, which is
obtained by aligning the sampling distribution to the real distribution.
Extensive experiments show that our plug-in design can be trained efficiently
and boost the inference performance of various state-of-the-art acceleration
methods, especially when there are few denoising steps. For example, when using
10 denoising steps on the popular LSUN Bedroom dataset, we improve the FID of
DDIM from 9.65 to 6.07, simply by adopting our method for a more appropriate
set of timesteps. Code will be made publicly available.
