  Despite the progress made in the style transfer task, most previous work
focus on transferring only relatively simple features like color or texture,
while missing more abstract concepts such as overall art expression or
painter-specific traits. However, these abstract semantics can be captured by
models like DALL-E or CLIP, which have been trained using huge datasets of
images and textual documents. In this paper, we propose StylerDALLE, a style
transfer method that exploits both of these models and uses natural language to
describe abstract art styles. Specifically, we formulate the language-guided
style transfer task as a non-autoregressive token sequence translation, i.e.,
from input content image to output stylized image, in the discrete latent space
of a large-scale pretrained vector-quantized tokenizer, e.g., the discrete
variational auto-encoder (dVAE) of DALL-E. To incorporate style information, we
propose a Reinforcement Learning strategy with CLIP-based language supervision
that ensures stylization and content preservation simultaneously. Experimental
results demonstrate the superiority of our method, which can effectively
transfer art styles using language instructions at different granularities.
Code is available at https://github.com/zipengxuc/StylerDALLE.
