  Extensive empirical evidence demonstrates that conditional generative models
are easier to train and perform better than unconditional ones by exploiting
the labels of data. So do score-based diffusion models. In this paper, we
analyze the phenomenon formally and identify that the key of conditional
learning is to partition the data properly. Inspired by the analyses, we
propose self-conditioned diffusion models (SCDM), which is trained conditioned
on indices clustered by the k-means algorithm on the features extracted by a
model pre-trained in a self-supervised manner. SCDM significantly improves the
unconditional model across various datasets and achieves a record-breaking FID
of 3.94 on ImageNet 64x64 without labels. Besides, SCDM achieves a slightly
better FID than the corresponding conditional model on CIFAR10.
