  Despite the recent advances in large-scale diffusion models, little progress
has been made on the layout-to-image (L2I) synthesis task. Current L2I models
either suffer from poor editability via text or weak alignment between the
generated image and the input layout. This limits their usability in practice.
To mitigate this, we propose to integrate adversarial supervision into the
conventional training pipeline of L2I diffusion models (ALDM). Specifically, we
employ a segmentation-based discriminator which provides explicit feedback to
the diffusion generator on the pixel-level alignment between the denoised image
and the input layout. To encourage consistent adherence to the input layout
over the sampling steps, we further introduce the multistep unrolling strategy.
Instead of looking at a single timestep, we unroll a few steps recursively to
imitate the inference process, and ask the discriminator to assess the
alignment of denoised images with the layout over a certain time window. Our
experiments show that ALDM enables layout faithfulness of the generated images,
while allowing broad editability via text prompts. Moreover, we showcase its
usefulness for practical applications: by synthesizing target distribution
samples via text control, we improve domain generalization of semantic
segmentation models by a large margin (~12 mIoU points).
