  Most diffusion models assume that the reverse process adheres to a Gaussian
distribution. However, this approximation has not been rigorously validated,
especially at singularities, where t=0 and t=1. Improperly dealing with such
singularities leads to an average brightness issue in applications, and limits
the generation of images with extreme brightness or darkness. We primarily
focus on tackling singularities from both theoretical and practical
perspectives. Initially, we establish the error bounds for the reverse process
approximation, and showcase its Gaussian characteristics at singularity time
steps. Based on this theoretical insight, we confirm the singularity at t=1 is
conditionally removable while it at t=0 is an inherent property. Upon these
significant conclusions, we propose a novel plug-and-play method SingDiffusion
to address the initial singular time step sampling, which not only effectively
resolves the average brightness issue for a wide range of diffusion models
without extra training efforts, but also enhances their generation capability
in achieving notable lower FID scores.
