  While conditional diffusion models are known to have good coverage of the
data distribution, they still face limitations in output diversity,
particularly when sampled with a high classifier-free guidance scale for
optimal image quality or when trained on small datasets. We attribute this
problem to the role of the conditioning signal in inference and offer an
improved sampling strategy for diffusion models that can increase generation
diversity, especially at high guidance scales, with minimal loss of sample
quality. Our sampling strategy anneals the conditioning signal by adding
scheduled, monotonically decreasing Gaussian noise to the conditioning vector
during inference to balance diversity and condition alignment. Our
Condition-Annealed Diffusion Sampler (CADS) can be used with any pretrained
model and sampling algorithm, and we show that it boosts the diversity of
diffusion models in various conditional generation tasks. Further, using an
existing pretrained diffusion model, CADS achieves a new state-of-the-art FID
of 1.70 and 2.31 for class-conditional ImageNet generation at 256$\times$256
and 512$\times$512 respectively.
