  Style transfer methods produce a transferred image which is a rendering of a
content image in the manner of a style image. There is a rich literature of
variant methods. However, evaluation procedures are qualitative, mostly
involving user studies. We describe a novel quantitative evaluation procedure.
One plots effectiveness (a measure of the extent to which the style was
transferred) against coherence (a measure of the extent to which the
transferred image decomposes into objects in the same way that the content
image does) to obtain an EC plot.
  We construct EC plots comparing a number of recent style transfer methods.
Most methods control within-layer gram matrices, but we also investigate a
method that controls cross-layer gram matrices. These EC plots reveal a number
of intriguing properties of recent style transfer methods. The style used has a
strong effect on the outcome, for all methods. Using large style weights does
not necessarily improve effectiveness, and can produce worse results.
Cross-layer gram matrices easily beat all other methods, but some styles remain
difficult for all methods. Ensemble methods show real promise. It is likely
that, for current methods, each style requires a different choice of weights to
obtain the best results, so that automated weight setting methods are
desirable. Finally, we show evidence comparing our EC evaluations to human
evaluations.
