  Realistic image manipulation is challenging because it requires modifying the
image appearance in a user-controlled way, while preserving the realism of the
result. Unless the user has considerable artistic skill, it is easy to "fall
off" the manifold of natural images while editing. In this paper, we propose to
learn the natural image manifold directly from data using a generative
adversarial neural network. We then define a class of image editing operations,
and constrain their output to lie on that learned manifold at all times. The
model automatically adjusts the output keeping all edits as realistic as
possible. All our manipulations are expressed in terms of constrained
optimization and are applied in near-real time. We evaluate our algorithm on
the task of realistic photo manipulation of shape and color. The presented
method can further be used for changing one image to look like the other, as
well as generating novel imagery from scratch based on user's scribbles.
