  We seek to give users precise control over diffusion-based image generation
by modeling complex scenes as sequences of layers, which define the desired
spatial arrangement and visual attributes of objects in the scene. Collage
Diffusion harmonizes the input layers to make objects fit together -- the key
challenge involves minimizing changes in the positions and key visual
attributes of the input layers while allowing other attributes to change in the
harmonization process. We ensure that objects are generated in the correct
locations by modifying text-image cross-attention with the layers' alpha masks.
We preserve key visual attributes of input layers by learning specialized text
representations per layer and by extending ControlNet to operate on layers.
Layer input allows users to control the extent of image harmonization on a
per-object basis, and users can even iteratively edit individual objects in
generated images while keeping other objects fixed. By leveraging the rich
information present in layer input, Collage Diffusion generates globally
harmonized images that maintain desired object characteristics better than
prior approaches.
