  Humans can envision a realistic photo given a free-hand sketch that is not
only spatially imprecise and geometrically distorted but also without colors
and visual details. We study unsupervised sketch-to-photo synthesis for the
first time, learning from unpaired sketch-photo data where the target photo for
a sketch is unknown during training. Existing works only deal with style change
or spatial deformation alone, synthesizing photos from edge-aligned line
drawings or transforming shapes within the same modality, e.g., color images.
Our key insight is to decompose unsupervised sketch-to-photo synthesis into a
two-stage translation task: First shape translation from sketches to grayscale
photos and then content enrichment from grayscale to color photos. We also
incorporate a self-supervised denoising objective and an attention module to
handle abstraction and style variations that are inherent and specific to
sketches. Our synthesis is sketch-faithful and photo-realistic to enable
sketch-based image retrieval in practice. An exciting corollary product is a
universal and promising sketch generator that captures human visual perception
beyond the edge map of a photo.
