  Controllable image synthesis with user scribbles has gained huge public
interest with the recent advent of text-conditioned latent diffusion models.
The user scribbles control the color composition while the text prompt provides
control over the overall image semantics. However, we note that prior works in
this direction suffer from an intrinsic domain shift problem, wherein the
generated outputs often lack details and resemble simplistic representations of
the target domain. In this paper, we propose a novel guided image synthesis
framework, which addresses this problem by modeling the output image as the
solution of a constrained optimization problem. We show that while computing an
exact solution to the optimization is infeasible, an approximation of the same
can be achieved while just requiring a single pass of the reverse diffusion
process. Additionally, we show that by simply defining a cross-attention based
correspondence between the input text tokens and the user stroke-painting, the
user is also able to control the semantics of different painted regions without
requiring any conditional training or finetuning. Human user study results show
that the proposed approach outperforms the previous state-of-the-art by over
85.32% on the overall user satisfaction scores. Project page for our paper is
available at https://1jsingh.github.io/gradop.
