  Style transfer is a useful image synthesis technique that can re-render given
image into another artistic style while preserving its content information.
Generative Adversarial Network (GAN) is a widely adopted framework toward this
task for its better representation ability on local style patterns than the
traditional Gram-matrix based methods. However, most previous methods rely on
sufficient amount of pre-collected style images to train the model. In this
paper, a novel Patch Permutation GAN (P$^2$-GAN) network that can efficiently
learn the stroke style from a single style image is proposed. We use patch
permutation to generate multiple training samples from the given style image. A
patch discriminator that can simultaneously process patch-wise images and
natural images seamlessly is designed. We also propose a local texture
descriptor based criterion to quantitatively evaluate the style transfer
quality. Experimental results showed that our method can produce finer quality
re-renderings from single style image with improved computational efficiency
compared with many state-of-the-arts methods.
