  In this work, we investigate an intriguing and prevalent phenomenon of
diffusion models which we term as "consistent model reproducibility": given the
same starting noise input and a deterministic sampler, different diffusion
models often yield remarkably similar outputs. We confirm this phenomenon
through comprehensive experiments, implying that different diffusion models
consistently reach the same data distribution and scoring function regardless
of diffusion model frameworks, model architectures, or training procedures.
More strikingly, our further investigation implies that diffusion models are
learning distinct distributions affected by the training data size. This is
supported by the fact that the model reproducibility manifests in two distinct
training regimes: (i) "memorization regime", where the diffusion model overfits
to the training data distribution, and (ii) "generalization regime", where the
model learns the underlying data distribution. Our study also finds that this
valuable property generalizes to many variants of diffusion models, including
those for conditional use, solving inverse problems, and model fine-tuning.
Finally, our work raises numerous intriguing theoretical questions for future
investigation and highlights practical implications regarding training
efficiency, model privacy, and the controlled generation of diffusion models.
