  Diffusion models have the ability to generate high quality images by
denoising pure Gaussian noise images. While previous research has primarily
focused on improving the control of image generation through adjusting the
denoising process, we propose a novel direction of manipulating the initial
noise to control the generated image. Through experiments on stable diffusion,
we show that blocks of pixels in the initial latent images have a preference
for generating specific content, and that modifying these blocks can
significantly influence the generated image. In particular, we show that
modifying a part of the initial image affects the corresponding region of the
generated image while leaving other regions unaffected, which is useful for
repainting tasks. Furthermore, we find that the generation preferences of pixel
blocks are primarily determined by their values, rather than their position. By
moving pixel blocks with a tendency to generate user-desired content to
user-specified regions, our approach achieves state-of-the-art performance in
layout-to-image generation. Our results highlight the flexibility and power of
initial image manipulation in controlling the generated image.
