  Controllable image synthesis with user scribbles is a topic of keen interest
in the computer vision community. In this paper, for the first time we study
the problem of photorealistic image synthesis from incomplete and primitive
human paintings. In particular, we propose a novel approach paint2pix, which
learns to predict (and adapt) "what a user wants to draw" from rudimentary
brushstroke inputs, by learning a mapping from the manifold of incomplete human
paintings to their realistic renderings. When used in conjunction with recent
works in autonomous painting agents, we show that paint2pix can be used for
progressive image synthesis from scratch. During this process, paint2pix allows
a novice user to progressively synthesize the desired image output, while
requiring just few coarse user scribbles to accurately steer the trajectory of
the synthesis process. Furthermore, we find that our approach also forms a
surprisingly convenient approach for real image editing, and allows the user to
perform a diverse range of custom fine-grained edits through the addition of
only a few well-placed brushstrokes. Supplemental video and demo are available
at https://1jsingh.github.io/paint2pix
