  Recent score-based diffusion models (SBDMs) show promising results in
unpaired image-to-image translation (I2I). However, existing methods, either
energy-based or statistically-based, provide no explicit form of the interfered
intermediate generative distributions. This work presents a new
score-decomposed diffusion model (SDDM) on manifolds to explicitly optimize the
tangled distributions during image generation. SDDM derives manifolds to make
the distributions of adjacent time steps separable and decompose the score
function or energy guidance into an image ``denoising" part and a content
``refinement" part. To refine the image in the same noise level, we equalize
the refinement parts of the score function and energy guidance, which permits
multi-objective optimization on the manifold. We also leverage the block
adaptive instance normalization module to construct manifolds with lower
dimensions but still concentrated with the perturbed reference image. SDDM
outperforms existing SBDM-based methods with much fewer diffusion steps on
several I2I benchmarks.
