  We propose a new flexible deep convolutional neural network (convnet) to
perform fast visual style transfer. In contrast to existing convnets that
address the same task, our architecture derives directly from the structure of
the gradient descent originally used to solve the style transfer problem [Gatys
et al., 2016]. Like existing convnets, ours approximately solves the original
problem much faster than the gradient descent. However, our network is uniquely
flexible by design: it can be manipulated at runtime to enforce new constraints
on the final solution. In particular, we show how to modify it to obtain a
photorealistic result with no retraining. We study the modifications made by
[Luan et al., 2017] to the original cost function of [Gatys et al., 2016] to
achieve photorealistic style transfer. These modifications affect directly the
gradient descent and can be reported on-the-fly in our network. These
modifications are possible as the proposed architecture stems from unrolling
the gradient descent.
