  Diffusion models are the current state-of-the-art in image generation,
synthesizing high-quality images by breaking down the generation process into
many fine-grained denoising steps. Despite their good performance, diffusion
models are computationally expensive, requiring many neural function
evaluations (NFEs). In this work, we propose an anytime diffusion-based method
that can generate viable images when stopped at arbitrary times before
completion. Using existing pretrained diffusion models, we show that the
generation scheme can be recomposed as two nested diffusion processes, enabling
fast iterative refinement of a generated image. In experiments on ImageNet and
Stable Diffusion-based text-to-image generation, we show, both qualitatively
and quantitatively, that our method's intermediate generation quality greatly
exceeds that of the original diffusion model, while the final generation result
remains comparable. We illustrate the applicability of Nested Diffusion in
several settings, including for solving inverse problems, and for rapid
text-based content creation by allowing user intervention throughout the
sampling process.
