  Iterative denoising-based generation, also known as denoising diffusion
models, has recently been shown to be comparable in quality to other classes of
generative models, and even surpass them. Including, in particular, Generative
Adversarial Networks, which are currently the state of the art in many
sub-tasks of image generation. However, a major drawback of this method is that
it requires hundreds of iterations to produce a competitive result. Recent
works have proposed solutions that allow for faster generation with fewer
iterations, but the image quality gradually deteriorates with increasingly
fewer iterations being applied during generation. In this paper, we reveal some
of the causes that affect the generation quality of diffusion models,
especially when sampling with few iterations, and come up with a simple, yet
effective, solution to mitigate them. We consider two opposite equations for
the iterative denoising, the first predicts the applied noise, and the second
predicts the image directly. Our solution takes the two options and learns to
dynamically alternate between them through the denoising process. Our proposed
solution is general and can be applied to any existing diffusion model. As we
show, when applied to various SOTA architectures, our solution immediately
improves their generation quality, with negligible added complexity and
parameters. We experiment on multiple datasets and configurations and run an
extensive ablation study to support these findings.
