  Diffusion models have exhibited impressive prowess in the text-to-image task.
Recent methods add image-level controls, e.g., edge and depth maps, to
manipulate the generation process together with text prompts to obtain desired
images. This controlling process is globally operated on the entire image,
which limits the flexibility of control regions. In this paper, we introduce a
new simple yet practical task setting: local control. It focuses on controlling
specific local areas according to user-defined image conditions, where the rest
areas are only conditioned by the original text prompt. This manner allows the
users to flexibly control the image generation in a fine-grained way. However,
it is non-trivial to achieve this goal. The naive manner of directly adding
local conditions may lead to the local control dominance problem. To mitigate
this problem, we propose a training-free method that leverages the updates of
noised latents and parameters in the cross-attention map during the denosing
process to promote concept generation in non-control areas. Moreover, we use
feature mask constraints to mitigate the degradation of synthesized image
quality caused by information differences inside and outside the local control
area. Extensive experiments demonstrate that our method can synthesize
high-quality images to the prompt under local control conditions. Code is
available at https://github.com/YibooZhao/Local-Control.
