  We propose a method for scene-level sketch-to-photo synthesis with text
guidance. Although object-level sketch-to-photo synthesis has been widely
studied, whole-scene synthesis is still challenging without reference photos
that adequately reflect the target style. To this end, we leverage knowledge
from recent large-scale pre-trained generative models, resulting in text-guided
sketch-to-photo synthesis without the need for reference images. To train our
model, we use self-supervised learning from a set of photographs. Specifically,
we use a pre-trained edge detector that maps both color and sketch images into
a standardized edge domain, which reduces the gap between photograph-based edge
images (during training) and hand-drawn sketch images (during inference). We
implement our method by fine-tuning a latent diffusion model (i.e., Stable
Diffusion) with sketch and text conditions. Experiments show that the proposed
method translates original sketch images that are not extracted from color
images into photos with compelling visual quality.
