  Recent advances in diffusion models attempt to handle conditional generative
tasks by utilizing a differentiable loss function for guidance without the need
for additional training. While these methods achieved certain success, they
often compromise on sample quality and require small guidance step sizes,
leading to longer sampling processes. This paper reveals that the fundamental
issue lies in the manifold deviation during the sampling process when loss
guidance is employed. We theoretically show the existence of manifold deviation
by establishing a certain lower bound for the estimation error of the loss
guidance. To mitigate this problem, we propose Diffusion with Spherical
Gaussian constraint (DSG), drawing inspiration from the concentration
phenomenon in high-dimensional Gaussian distributions. DSG effectively
constrains the guidance step within the intermediate data manifold through
optimization and enables the use of larger guidance steps. Furthermore, we
present a closed-form solution for DSG denoising with the Spherical Gaussian
constraint. Notably, DSG can seamlessly integrate as a plugin module within
existing training-free conditional diffusion methods. Implementing DSG merely
involves a few lines of additional code with almost no extra computational
overhead, yet it leads to significant performance improvements. Comprehensive
experimental results in various conditional generation tasks validate the
superiority and adaptability of DSG in terms of both sample quality and time
efficiency.
