  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable
success in various image generation tasks compared with Generative Adversarial
Nets (GANs). Recent work on semantic image synthesis mainly follows the
\emph{de facto} GAN-based approaches, which may lead to unsatisfactory quality
or diversity of generated images. In this paper, we propose a novel framework
based on DDPM for semantic image synthesis. Unlike previous conditional
diffusion model directly feeds the semantic layout and noisy image as input to
a U-Net structure, which may not fully leverage the information in the input
semantic mask, our framework processes semantic layout and noisy image
differently. It feeds noisy image to the encoder of the U-Net structure while
the semantic layout to the decoder by multi-layer spatially-adaptive
normalization operators. To further improve the generation quality and semantic
interpretability in semantic image synthesis, we introduce the classifier-free
guidance sampling strategy, which acknowledge the scores of an unconditional
model for sampling process. Extensive experiments on three benchmark datasets
demonstrate the effectiveness of our proposed method, achieving
state-of-the-art performance in terms of fidelity (FID) and diversity (LPIPS).
