  Diffusion-based image translation guided by semantic texts or a single target
image has enabled flexible style transfer which is not limited to the specific
domains. Unfortunately, due to the stochastic nature of diffusion models, it is
often difficult to maintain the original content of the image during the
reverse diffusion. To address this, here we present a novel diffusion-based
unsupervised image translation method using disentangled style and content
representation.
  Specifically, inspired by the splicing Vision Transformer, we extract
intermediate keys of multihead self attention layer from ViT model and used
them as the content preservation loss. Then, an image guided style transfer is
performed by matching the [CLS] classification token from the denoised samples
and target image, whereas additional CLIP loss is used for the text-driven
style transfer. To further accelerate the semantic change during the reverse
diffusion, we also propose a novel semantic divergence loss and resampling
strategy. Our experimental results show that the proposed method outperforms
state-of-the-art baseline models in both text-guided and image-guided
translation tasks.
