  Universal style transfer aims to transfer arbitrary visual styles to content
images. Existing feed-forward based methods, while enjoying the inference
efficiency, are mainly limited by inability of generalizing to unseen styles or
compromised visual quality. In this paper, we present a simple yet effective
method that tackles these limitations without training on any pre-defined
styles. The key ingredient of our method is a pair of feature transforms,
whitening and coloring, that are embedded to an image reconstruction network.
The whitening and coloring transforms reflect a direct matching of feature
covariance of the content image to a given style image, which shares similar
spirits with the optimization of Gram matrix based cost in neural style
transfer. We demonstrate the effectiveness of our algorithm by generating
high-quality stylized images with comparisons to a number of recent methods. We
also analyze our method by visualizing the whitened features and synthesizing
textures via simple feature coloring.
