  The primary axes of interest in image-generating diffusion models are image
quality, the amount of variation in the results, and how well the results align
with a given condition, e.g., a class label or a text prompt. The popular
classifier-free guidance approach uses an unconditional model to guide a
conditional model, leading to simultaneously better prompt alignment and
higher-quality images at the cost of reduced variation. These effects seem
inherently entangled, and thus hard to control. We make the surprising
observation that it is possible to obtain disentangled control over image
quality without compromising the amount of variation by guiding generation
using a smaller, less-trained version of the model itself rather than an
unconditional model. This leads to significant improvements in ImageNet
generation, setting record FIDs of 1.01 for 64x64 and 1.25 for 512x512, using
publicly available networks. Furthermore, the method is also applicable to
unconditional diffusion models, drastically improving their quality.
