  Many problems in image processing and computer vision (e.g. colorization,
style transfer) can be posed as 'manipulating' an input image into a
corresponding output image given a user-specified guiding signal. A holy-grail
solution towards generic image manipulation should be able to efficiently alter
an input image with any personalized signals (even signals unseen during
training), such as diverse paintings and arbitrary descriptive attributes.
However, existing methods are either inefficient to simultaneously process
multiple signals (let alone generalize to unseen signals), or unable to handle
signals from other modalities. In this paper, we make the first attempt to
address the zero-shot image manipulation task. We cast this problem as
manipulating an input image according to a parametric model whose key
parameters can be conditionally generated from any guiding signal (even unseen
ones). To this end, we propose the Zero-shot Manipulation Net (ZM-Net), a
fully-differentiable architecture that jointly optimizes an
image-transformation network (TNet) and a parameter network (PNet). The PNet
learns to generate key transformation parameters for the TNet given any guiding
signal while the TNet performs fast zero-shot image manipulation according to
both signal-dependent parameters from the PNet and signal-invariant parameters
from the TNet itself. Extensive experiments show that our ZM-Net can perform
high-quality image manipulation conditioned on different forms of guiding
signals (e.g. style images and attributes) in real-time (tens of milliseconds
per image) even for unseen signals. Moreover, a large-scale style dataset with
over 20,000 style images is also constructed to promote further research.
