  AI illustrator aims to automatically design visually appealing images for
books to provoke rich thoughts and emotions. To achieve this goal, we propose a
framework for translating raw descriptions with complex semantics into
semantically corresponding images. The main challenge lies in the complexity of
the semantics of raw descriptions, which may be hard to be visualized (e.g.,
"gloomy" or "Asian"). It usually poses challenges for existing methods to
handle such descriptions. To address this issue, we propose a Prompt-based
Cross-Modal Generation Framework (PCM-Frame) to leverage two powerful
pre-trained models, including CLIP and StyleGAN. Our framework consists of two
components: a projection module from Text Embeddings to Image Embeddings based
on prompts, and an adapted image generation module built on StyleGAN which
takes Image Embeddings as inputs and is trained by combined semantic
consistency losses. To bridge the gap between realistic images and illustration
designs, we further adopt a stylization model as post-processing in our
framework for better visual effects. Benefiting from the pre-trained models,
our method can handle complex descriptions and does not require external paired
data for training. Furthermore, we have built a benchmark that consists of 200
raw descriptions. We conduct a user study to demonstrate our superiority over
the competing methods with complicated texts. We release our code at
https://github.com/researchmm/AI_Illustrator.
